Why does anyone use Python?

(written by lawrence krubner, however indented passages are often quotes). You can contact lawrence at: lawrence@krubner.com, or follow me on Twitter.

Summary: Python 1.x and 2.x brought some interesting ideas to the tech world, but the upgrade to 3.x has brought an astonishing number of problems in terms of version conflicts and dependency management. Python no longer offers ease of setup, as any non-trivial system will run into version problems. If you are a CTO, you should consider using a technology stack that has a long history of being stable, such as any of the languages that run on the JVM and compile to Java bytecode.

Over the last 20 years I’ve written in software in PHP and Ruby and Javascript and Java and Swift and Scala and Perl and Coffeescript and TypeScript and Clojure and Python. My favorite is Clojure. Python is competitive in the race for “most annoying eco-system.”

Despite the fact that PHP has some hilarious essays about how broken it is I used to enjoy writing PHP code, especially PHP 4, before they added unnecessary Object Oriented junk in PHP 5. I am happy to praise PHP for having a dependency management model that mostly works.

But Python? It seems like everything is broken in Python all the time. I’ve written about this a few times lately, and I’m just struck, once again, tonight, how the simplest and most obvious things send me spiraling into lost hours, trying to figure out why some command isn’t working.

So for instance, tonight, the intern who is working for us for the summer said “Can you help me connect my Django app to the web?” and I said “sure”. I looked around for a tutorial about attaching Django to Nginx, and I found this:

https://uwsgi-docs.readthedocs.io/en/latest/tutorials/Django_and_nginx.html

Which tells me to do this:

pip install uwsgi

and I get:

pip install uwsgi
Collecting uwsgi
  Using cached https://files.pythonhosted.org/packages/a2/c9/a2d5737f63cd9df4317a4acc15d1ddf4952e28398601d8d7d706c16381e0/uwsgi-2.0.17.1.tar.gz
Installing collected packages: uwsgi
  Running setup.py install for uwsgi ... error
    Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-wg1rqgn8/uwsgi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-r99m_baa-record/install-record.txt --single-version-externally-managed --compile:
    /usr/lib64/python3.7/distutils/dist.py:274: UserWarning: Unknown distribution option: 'descriptions'
      warnings.warn(msg)
    running install
    using profile: buildconf/default.ini
    detected include path: ['/usr/lib/gcc/x86_64-redhat-linux/7/include', '/usr/local/include', '/usr/include']
    Patching "bin_name" to properly install_scripts dir
    detected CPU cores: 2
    configured CFLAGS: -O2 -I. -Wall -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -DUWSGI_HAS_IFADDRS -DUWSGI_ZLIB -DUWSGI_LOCK_USE_MUTEX -DUWSGI_EVENT_USE_EPOLL -DUWSGI_EVENT_TIMER_USE_TIMERFD -DUWSGI_EVENT_FILEMONITOR_USE_INOTIFY -DUWSGI_VERSION="\"2.0.17.1\"" -DUWSGI_VERSION_BASE="2" -DUWSGI_VERSION_MAJOR="0" -DUWSGI_VERSION_MINOR="17" -DUWSGI_VERSION_REVISION="1" -DUWSGI_VERSION_CUSTOM="\"\"" -DUWSGI_YAML -DUWSGI_PLUGIN_DIR="\".\"" -DUWSGI_DECLARE_EMBEDDED_PLUGINS="UDEP(python);UDEP(gevent);UDEP(ping);UDEP(cache);UDEP(nagios);UDEP(rrdtool);UDEP(carbon);UDEP(rpc);UDEP(corerouter);UDEP(fastrouter);UDEP(http);UDEP(ugreen);UDEP(signal);UDEP(syslog);UDEP(rsyslog);UDEP(logsocket);UDEP(router_uwsgi);UDEP(router_redirect);UDEP(router_basicauth);UDEP(zergpool);UDEP(redislog);UDEP(mongodblog);UDEP(router_rewrite);UDEP(router_http);UDEP(logfile);UDEP(router_cache);UDEP(rawrouter);UDEP(router_static);UDEP(sslrouter);UDEP(spooler);UDEP(cheaper_busyness);UDEP(symcall);UDEP(transformation_tofile);UDEP(transformation_gzip);UDEP(transformation_chunked);UDEP(transformation_offload);UDEP(router_memcached);UDEP(router_redis);UDEP(router_hash);UDEP(router_expires);UDEP(router_metrics);UDEP(transformation_template);UDEP(stats_pusher_socket);" -DUWSGI_LOAD_EMBEDDED_PLUGINS="ULEP(python);ULEP(gevent);ULEP(ping);ULEP(cache);ULEP(nagios);ULEP(rrdtool);ULEP(carbon);ULEP(rpc);ULEP(corerouter);ULEP(fastrouter);ULEP(http);ULEP(ugreen);ULEP(signal);ULEP(syslog);ULEP(rsyslog);ULEP(logsocket);ULEP(router_uwsgi);ULEP(router_redirect);ULEP(router_basicauth);ULEP(zergpool);ULEP(redislog);ULEP(mongodblog);ULEP(router_rewrite);ULEP(router_http);ULEP(logfile);ULEP(router_cache);ULEP(rawrouter);ULEP(router_static);ULEP(sslrouter);ULEP(spooler);ULEP(cheaper_busyness);ULEP(symcall);ULEP(transformation_tofile);ULEP(transformation_gzip);ULEP(transformation_chunked);ULEP(transformation_offload);ULEP(router_memcached);ULEP(router_redis);ULEP(router_hash);ULEP(router_expires);ULEP(router_metrics);ULEP(transformation_template);ULEP(stats_pusher_socket);"
    *** uWSGI compiling server core ***
    [thread 1][gcc -pthread] core/utils.o
    [thread 0][gcc -pthread] core/protocol.o
    [thread 0][gcc -pthread] core/socket.o
    [thread 0][gcc -pthread] core/logging.o
    [thread 1][gcc -pthread] core/master.o
    [thread 0][gcc -pthread] core/master_utils.o
    [thread 1][gcc -pthread] core/emperor.o
    [thread 0][gcc -pthread] core/notify.o
    [thread 0][gcc -pthread] core/mule.o
    [thread 1][gcc -pthread] core/subscription.o
    [thread 0][gcc -pthread] core/stats.o
    [thread 1][gcc -pthread] core/sendfile.o
    [thread 1][gcc -pthread] core/async.o
    [thread 0][gcc -pthread] core/master_checks.o
    [thread 1][gcc -pthread] core/fifo.o
    [thread 0][gcc -pthread] core/offload.o
    [thread 1][gcc -pthread] core/io.o
    [thread 0][gcc -pthread] core/static.o
    [thread 0][gcc -pthread] core/websockets.o
    [thread 1][gcc -pthread] core/spooler.o
    [thread 0][gcc -pthread] core/snmp.o
    [thread 0][gcc -pthread] core/exceptions.o
    [thread 1][gcc -pthread] core/config.o
    [thread 0][gcc -pthread] core/setup_utils.o
    [thread 0][gcc -pthread] core/clock.o
    [thread 1][gcc -pthread] core/init.o
    [thread 0][gcc -pthread] core/buffer.o
    [thread 1][gcc -pthread] core/reader.o
    [thread 0][gcc -pthread] core/writer.o
    [thread 1][gcc -pthread] core/alarm.o
    [thread 0][gcc -pthread] core/cron.o
    [thread 1][gcc -pthread] core/hooks.o
    [thread 0][gcc -pthread] core/plugins.o
    [thread 1][gcc -pthread] core/lock.o
    [thread 0][gcc -pthread] core/cache.o
    [thread 1][gcc -pthread] core/daemons.o
    [thread 1][gcc -pthread] core/errors.o
    [thread 1][gcc -pthread] core/hash.o
    [thread 1][gcc -pthread] core/master_events.o
    [thread 0][gcc -pthread] core/chunked.o
    [thread 1][gcc -pthread] core/queue.o
    [thread 0][gcc -pthread] core/event.o
    [thread 1][gcc -pthread] core/signal.o
    [thread 0][gcc -pthread] core/strings.o
    [thread 1][gcc -pthread] core/progress.o
    [thread 0][gcc -pthread] core/timebomb.o
    [thread 1][gcc -pthread] core/ini.o
    [thread 0][gcc -pthread] core/fsmon.o
    [thread 1][gcc -pthread] core/mount.o
    [thread 0][gcc -pthread] core/metrics.o
    [thread 1][gcc -pthread] core/plugins_builder.o
    [thread 1][gcc -pthread] core/sharedarea.o
    [thread 0][gcc -pthread] core/rpc.o
    [thread 1][gcc -pthread] core/gateway.o
    [thread 0][gcc -pthread] core/loop.o
    [thread 1][gcc -pthread] core/cookie.o
    [thread 0][gcc -pthread] core/querystring.o
    [thread 1][gcc -pthread] core/rb_timers.o
    [thread 0][gcc -pthread] core/transformations.o
    [thread 1][gcc -pthread] core/uwsgi.o
    [thread 0][gcc -pthread] proto/base.o
    [thread 0][gcc -pthread] proto/uwsgi.o
    [thread 0][gcc -pthread] proto/http.o
    [thread 0][gcc -pthread] proto/fastcgi.o
    [thread 0][gcc -pthread] proto/scgi.o
    [thread 0][gcc -pthread] proto/puwsgi.o
    [thread 0][gcc -pthread] lib/linux_ns.o
    [thread 0][gcc -pthread] core/zlib.o
    [thread 1][gcc -pthread] core/yaml.o
    [thread 0][gcc -pthread] core/dot_h.o
    [thread 0][gcc -pthread] core/config_py.o
    *** uWSGI compiling embedded plugins ***
    [thread 0][gcc -pthread] plugins/python/python_plugin.o
    [thread 1][gcc -pthread] plugins/python/pyutils.o
    [thread 1][gcc -pthread] plugins/python/pyloader.o
    [thread 1][gcc -pthread] plugins/python/wsgi_handlers.o
    [thread 0][gcc -pthread] plugins/python/wsgi_headers.o
    [thread 0][gcc -pthread] plugins/python/wsgi_subhandler.o
    [thread 1][gcc -pthread] plugins/python/web3_subhandler.o
    [thread 0][gcc -pthread] plugins/python/pump_subhandler.o
    [thread 1][gcc -pthread] plugins/python/gil.o
    [thread 1][gcc -pthread] plugins/python/uwsgi_pymodule.o
    [thread 0][gcc -pthread] plugins/python/profiler.o
    [thread 0][gcc -pthread] plugins/python/symimporter.o
    [thread 0][gcc -pthread] plugins/python/tracebacker.o
    [thread 0][gcc -pthread] plugins/python/raw.o
    [thread 0][gcc -pthread] plugins/gevent/gevent.o
    [thread 0][gcc -pthread] plugins/gevent/hooks.o
    [thread 1][gcc -pthread] plugins/ping/ping_plugin.o
    [thread 0][gcc -pthread] plugins/cache/cache.o
    [thread 1][gcc -pthread] plugins/nagios/nagios.o
    [thread 1][gcc -pthread] plugins/rrdtool/rrdtool.o
    [thread 0][gcc -pthread] plugins/carbon/carbon.o
    [thread 1][gcc -pthread] plugins/rpc/rpc_plugin.o
    [thread 0][gcc -pthread] plugins/corerouter/cr_common.o
    [thread 1][gcc -pthread] plugins/corerouter/cr_map.o
    [thread 1][gcc -pthread] plugins/corerouter/corerouter.o
    [thread 0][gcc -pthread] plugins/fastrouter/fastrouter.o
    [thread 0][gcc -pthread] plugins/http/http.o
    [thread 1][gcc -pthread] plugins/http/keepalive.o
    [thread 1][gcc -pthread] plugins/http/https.o
    [thread 1][gcc -pthread] plugins/http/spdy3.o
    [thread 1][gcc -pthread] plugins/ugreen/ugreen.o
    [thread 0][gcc -pthread] plugins/signal/signal_plugin.o
    [thread 1][gcc -pthread] plugins/syslog/syslog_plugin.o
    [thread 0][gcc -pthread] plugins/rsyslog/rsyslog_plugin.o
    [thread 1][gcc -pthread] plugins/logsocket/logsocket_plugin.o
    [thread 0][gcc -pthread] plugins/router_uwsgi/router_uwsgi.o
    [thread 1][gcc -pthread] plugins/router_redirect/router_redirect.o
    [thread 0][gcc -pthread] plugins/router_basicauth/router_basicauth.o
    [thread 1][gcc -pthread] plugins/zergpool/zergpool.o
    [thread 0][gcc -pthread] plugins/redislog/redislog_plugin.o
    [thread 1][gcc -pthread] plugins/mongodblog/mongodblog_plugin.o
    [thread 0][gcc -pthread] plugins/router_rewrite/router_rewrite.o
    [thread 1][gcc -pthread] plugins/router_http/router_http.o
    [thread 0][gcc -pthread] plugins/logfile/logfile.o
    [thread 1][gcc -pthread] plugins/router_cache/router_cache.o
    [thread 0][gcc -pthread] plugins/rawrouter/rawrouter.o
    [thread 1][gcc -pthread] plugins/router_static/router_static.o
    [thread 1][gcc -pthread] plugins/sslrouter/sslrouter.o
    [thread 1][gcc -pthread] plugins/spooler/spooler_plugin.o
    [thread 0][gcc -pthread] plugins/cheaper_busyness/cheaper_busyness.o
    [thread 1][gcc -pthread] plugins/symcall/symcall_plugin.o
    [thread 1][gcc -pthread] plugins/transformation_tofile/tofile.o
    [thread 0][gcc -pthread] plugins/transformation_gzip/gzip.o
    [thread 1][gcc -pthread] plugins/transformation_chunked/chunked.o
    [thread 0][gcc -pthread] plugins/transformation_offload/offload.o
    [thread 1][gcc -pthread] plugins/router_memcached/router_memcached.o
    [thread 0][gcc -pthread] plugins/router_redis/router_redis.o
    [thread 1][gcc -pthread] plugins/router_hash/router_hash.o
    [thread 0][gcc -pthread] plugins/router_expires/expires.o
    [thread 1][gcc -pthread] plugins/router_metrics/plugin.o
    [thread 0][gcc -pthread] plugins/transformation_template/tt.o
    [thread 1][gcc -pthread] plugins/stats_pusher_socket/plugin.o
    *** uWSGI linking ***
    gcc -pthread -o /usr/local/bin/uwsgi -L/usr/lib64 -Wl,-rpath,/usr/lib64 core/utils.o core/protocol.o core/socket.o core/logging.o core/master.o core/master_utils.o core/emperor.o core/notify.o core/mule.o core/subscription.o core/stats.o core/sendfile.o core/async.o core/master_checks.o core/fifo.o core/offload.o core/io.o core/static.o core/websockets.o core/spooler.o core/snmp.o core/exceptions.o core/config.o core/setup_utils.o core/clock.o core/init.o core/buffer.o core/reader.o core/writer.o core/alarm.o core/cron.o core/hooks.o core/plugins.o core/lock.o core/cache.o core/daemons.o core/errors.o core/hash.o core/master_events.o core/chunked.o core/queue.o core/event.o core/signal.o core/strings.o core/progress.o core/timebomb.o core/ini.o core/fsmon.o core/mount.o core/metrics.o core/plugins_builder.o core/sharedarea.o core/rpc.o core/gateway.o core/loop.o core/cookie.o core/querystring.o core/rb_timers.o core/transformations.o core/uwsgi.o proto/base.o proto/uwsgi.o proto/http.o proto/fastcgi.o proto/scgi.o proto/puwsgi.o lib/linux_ns.o core/zlib.o core/yaml.o core/dot_h.o core/config_py.o plugins/python/python_plugin.o plugins/python/pyutils.o plugins/python/pyloader.o plugins/python/wsgi_handlers.o plugins/python/wsgi_headers.o plugins/python/wsgi_subhandler.o plugins/python/web3_subhandler.o plugins/python/pump_subhandler.o plugins/python/gil.o plugins/python/uwsgi_pymodule.o plugins/python/profiler.o plugins/python/symimporter.o plugins/python/tracebacker.o plugins/python/raw.o plugins/gevent/gevent.o plugins/gevent/hooks.o plugins/ping/ping_plugin.o plugins/cache/cache.o plugins/nagios/nagios.o plugins/rrdtool/rrdtool.o plugins/carbon/carbon.o plugins/rpc/rpc_plugin.o plugins/corerouter/cr_common.o plugins/corerouter/cr_map.o plugins/corerouter/corerouter.o plugins/fastrouter/fastrouter.o plugins/http/http.o plugins/http/keepalive.o plugins/http/https.o plugins/http/spdy3.o plugins/ugreen/ugreen.o plugins/signal/signal_plugin.o plugins/syslog/syslog_plugin.o plugins/rsyslog/rsyslog_plugin.o plugins/logsocket/logsocket_plugin.o plugins/router_uwsgi/router_uwsgi.o plugins/router_redirect/router_redirect.o plugins/router_basicauth/router_basicauth.o plugins/zergpool/zergpool.o plugins/redislog/redislog_plugin.o plugins/mongodblog/mongodblog_plugin.o plugins/router_rewrite/router_rewrite.o plugins/router_http/router_http.o plugins/logfile/logfile.o plugins/router_cache/router_cache.o plugins/rawrouter/rawrouter.o plugins/router_static/router_static.o plugins/sslrouter/sslrouter.o plugins/spooler/spooler_plugin.o plugins/cheaper_busyness/cheaper_busyness.o plugins/symcall/symcall_plugin.o plugins/transformation_tofile/tofile.o plugins/transformation_gzip/gzip.o plugins/transformation_chunked/chunked.o plugins/transformation_offload/offload.o plugins/router_memcached/router_memcached.o plugins/router_redis/router_redis.o plugins/router_hash/router_hash.o plugins/router_expires/expires.o plugins/router_metrics/plugin.o plugins/transformation_template/tt.o plugins/stats_pusher_socket/plugin.o -lpthread -lm -rdynamic -ldl -lz -lpthread -ldl -lutil -lm -lpython3.7m -lcrypt
    /usr/bin/ld: cannot open output file /usr/local/bin/uwsgi: Permission denied
    collect2: error: ld returned 1 exit status
    *** error linking uWSGI ***
    
    ----------------------------------------
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-wg1rqgn8/uwsgi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-r99m_baa-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-wg1rqgn8/uwsgi/


—————
—————
—————

What’s amazing to me is how common this is with Python. In the world of Ruby, I can use gem and bundler. Once in a while I have a problem with something like nokogiri, but Ruby Bundler mostly works. And in the world of Clojure, I’ve never had a problem, because Leiningen is best build tool ever created.

So anyway, in Python, the most common reason things are broken is that Python3 is a completely separate language versus Python2, so I try:

pip3 install uwsgi

This gets me the same error.

I go looking for answers and I find:

https://github.com/GoogleCloudPlatform/python-runtime/issues/96

A curious fact: when I ask my co-workers “Why has Python gotten so popular over the last 10 years?” one of the main answers I’m given is “It’s really easy for beginners to get started”. And yet, for me, when I was a beginner, one of the main things I wanted was the ability to write code without having to dig deep into the internals of the system that I was using. Back in 2001, when I wrote PHP code, I could write all day without having to think too much about the way PHP connected to port 80 on the server. But in Python I have to think about it, and the dependency management system is broken, so I can’t just install the libraries I need.

I seriously do not get why the tech industry is sticking with Python. And not just sticking with it, but doubling down on it. As near as I can tell, Python is rising in popularity, even as PHP and Ruby die away.

“Easy for beginners” was Hypercard, and I loved writing stacks back in 1994, and the C extensions were easy to install. And 24 years later have passed, and “Easy for beginners” seems to have gotten much more difficult. How is that even possible? Why is the tech industry going in the wrong direction?

=================

[ [ UPDATE 2018-08-11 ] ]

Or how about this:

./manage.py runserver 0.0.0.0:8000

Which gives me:

Which leads me to this:

https://stackoverflow.com/questions/42611593/how-to-solve-syntaxerror-on-autogenerated-manage-py

The top 2 suggestions amount to “make sure you are running Django using the correct version of Python” so I try this:

Oh, wait, I see, maybe I just need to reinstall Django. Okay, let’s do that:

Compare this to PHP or Ruby or Javascript or Java or Scala or Clojure or literally any other language. There is simply no other popular technology that has such pervasive version conflicts.

I know I’ve complained, on this blog, about the times when Ruby Bundler can’t compile Nokogiri, but that seems minor compared to the version conflicts in Python.

————————————

————————————

[ [ UPDATE 2018-08-19 ] ]

More struggles with bizarre version conflicts. This:

/usr/local/lib/python3.7/site-packages/gunicorn/gunicorn ecommerce.wsgi:application –bind 0.0.0.0:8000

gives me:

ImportError: cannot import name ‘cookies’ from ‘http’ (/usr/local/lib/python3.7/site-packages/gunicorn/http/__init__.py)

Hmm, maybe this is a version conflict, so let’s upgrade everything.

I feel badly that I ever complained about dependency management in the world of Ruby. Whatever problems Ruby has, they seem like a dream compared to Python. I’ve really never worked with an eco-system where the dependency management is as broken as what I face in Python.

And there is also the fact that the number of conflicts are nearly infinite, and the systems seems creative in finding new combinations, so I don’t seem to be able to find my errors on Google.

——-
——-
——-

UPDATE 2018-08-21

Starting over again on a fresh EC2 instance, trying to do everything the “correct” way. Interesting that no matter what I do, there is always some error:

This leaves me with the impression that no one in the Python community is certain how to resolve all of these version conflicts.

——————–

——————–

——————–

Another observation about the culture of the community of Python programmers:

Most of the blog posts, tutorials, and StackOverflow posts suggest that one has to activate a virtual environment before installing something like gunicorn, but the official documentation creates the misleading illusion that installation is easy:

Of course, in the real world, you would never just do what you see in this screenshot. There would be several other steps.

Post external references

  1. 1
    https://eev.ee/blog/2012/04/09/php-a-fractal-of-bad-design/
  2. 2
    https://uwsgi-docs.readthedocs.io/en/latest/tutorials/Django_and_nginx.html
  3. 3
    https://leiningen.org/
  4. 4
    https://github.com/GoogleCloudPlatform/python-runtime/issues/96
  5. 5
    https://stackoverflow.com/questions/42611593/how-to-solve-syntaxerror-on-autogenerated-manage-py
  6. 6
    https://www.google.com/search?ei=NOd5W8rPENDs5gLtwLyYAw&q=ImportError%3A+cannot+import+name+%27Request%27+from+%27request%27&oq=ImportError%3A+cannot+import+name+%27Request%27+from+%27request%27&gs_l=psy-ab.3..35i39k1j0j0i8i30k1l3.12569.13485.0.14215.12.6.0.0.0.0.152.596.5j1.6.0....0...1c.1.64.psy-ab..10.1.151....0.VMvkr1VNio0
  7. 7
    http://docs.gunicorn.org/en/stable/install.html
Source