August 10th, 2018

# Why does anyone use Python?

(written by lawrence krubner, however indented passages are often quotes). You can contact lawrence at: lawrence@krubner.com

Over the last 20 years I’ve written in software in PHP and Ruby and Javascript and Java and Swift and Scala and Perl and Coffeescript and TypeScript and Clojure and Python. My favorite is Clojure. My least favorite is Scala, but Python is competitive in the race for “most annoying eco-system.”

Despite the fact that PHP has some hilarious essays about how broken it is I used to enjoy writing PHP code, especially PHP 4, before they added unnecessary Object Oriented junk in PHP 5. I am happy to praise PHP for having a dependency management model that mostly works.

But Python? It seems like everything is broken in Python all the time. I’ve written about this a few times lately, and I’m just struck, once again, tonight, how the simplest and most obvious things send me spiraling into lost hours, trying to figure out why some command isn’t working.

So for instance, tonight, the intern who is working for us for the summer said “Can you help me connect my Django app to the web?” and I said “sure”. I looked around for a tutorial about attaching Django to Nginx, and I found this:

Which tells me to do this:

pip install uwsgi

and I get:

pip install uwsgi
Collecting uwsgi
Using cached https://files.pythonhosted.org/packages/a2/c9/a2d5737f63cd9df4317a4acc15d1ddf4952e28398601d8d7d706c16381e0/uwsgi-2.0.17.1.tar.gz
Installing collected packages: uwsgi
Running setup.py install for uwsgi ... error
Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-wg1rqgn8/uwsgi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-r99m_baa-record/install-record.txt --single-version-externally-managed --compile:
/usr/lib64/python3.7/distutils/dist.py:274: UserWarning: Unknown distribution option: 'descriptions'
warnings.warn(msg)
running install
using profile: buildconf/default.ini
detected include path: ['/usr/lib/gcc/x86_64-redhat-linux/7/include', '/usr/local/include', '/usr/include']
Patching "bin_name" to properly install_scripts dir
detected CPU cores: 2
configured CFLAGS: -O2 -I. -Wall -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -DUWSGI_HAS_IFADDRS -DUWSGI_ZLIB -DUWSGI_LOCK_USE_MUTEX -DUWSGI_EVENT_USE_EPOLL -DUWSGI_EVENT_TIMER_USE_TIMERFD -DUWSGI_EVENT_FILEMONITOR_USE_INOTIFY -DUWSGI_VERSION="\"2.0.17.1\"" -DUWSGI_VERSION_BASE="2" -DUWSGI_VERSION_MAJOR="0" -DUWSGI_VERSION_MINOR="17" -DUWSGI_VERSION_REVISION="1" -DUWSGI_VERSION_CUSTOM="\"\"" -DUWSGI_YAML -DUWSGI_PLUGIN_DIR="\".\"" -DUWSGI_DECLARE_EMBEDDED_PLUGINS="UDEP(python);UDEP(gevent);UDEP(ping);UDEP(cache);UDEP(nagios);UDEP(rrdtool);UDEP(carbon);UDEP(rpc);UDEP(corerouter);UDEP(fastrouter);UDEP(http);UDEP(ugreen);UDEP(signal);UDEP(syslog);UDEP(rsyslog);UDEP(logsocket);UDEP(router_uwsgi);UDEP(router_redirect);UDEP(router_basicauth);UDEP(zergpool);UDEP(redislog);UDEP(mongodblog);UDEP(router_rewrite);UDEP(router_http);UDEP(logfile);UDEP(router_cache);UDEP(rawrouter);UDEP(router_static);UDEP(sslrouter);UDEP(spooler);UDEP(cheaper_busyness);UDEP(symcall);UDEP(transformation_tofile);UDEP(transformation_gzip);UDEP(transformation_chunked);UDEP(transformation_offload);UDEP(router_memcached);UDEP(router_redis);UDEP(router_hash);UDEP(router_expires);UDEP(router_metrics);UDEP(transformation_template);UDEP(stats_pusher_socket);" -DUWSGI_LOAD_EMBEDDED_PLUGINS="ULEP(python);ULEP(gevent);ULEP(ping);ULEP(cache);ULEP(nagios);ULEP(rrdtool);ULEP(carbon);ULEP(rpc);ULEP(corerouter);ULEP(fastrouter);ULEP(http);ULEP(ugreen);ULEP(signal);ULEP(syslog);ULEP(rsyslog);ULEP(logsocket);ULEP(router_uwsgi);ULEP(router_redirect);ULEP(router_basicauth);ULEP(zergpool);ULEP(redislog);ULEP(mongodblog);ULEP(router_rewrite);ULEP(router_http);ULEP(logfile);ULEP(router_cache);ULEP(rawrouter);ULEP(router_static);ULEP(sslrouter);ULEP(spooler);ULEP(cheaper_busyness);ULEP(symcall);ULEP(transformation_tofile);ULEP(transformation_gzip);ULEP(transformation_chunked);ULEP(transformation_offload);ULEP(router_memcached);ULEP(router_redis);ULEP(router_hash);ULEP(router_expires);ULEP(router_metrics);ULEP(transformation_template);ULEP(stats_pusher_socket);"
*** uWSGI compiling server core ***
*** uWSGI compiling embedded plugins ***
gcc -pthread -o /usr/local/bin/uwsgi -L/usr/lib64 -Wl,-rpath,/usr/lib64 core/utils.o core/protocol.o core/socket.o core/logging.o core/master.o core/master_utils.o core/emperor.o core/notify.o core/mule.o core/subscription.o core/stats.o core/sendfile.o core/async.o core/master_checks.o core/fifo.o core/offload.o core/io.o core/static.o core/websockets.o core/spooler.o core/snmp.o core/exceptions.o core/config.o core/setup_utils.o core/clock.o core/init.o core/buffer.o core/reader.o core/writer.o core/alarm.o core/cron.o core/hooks.o core/plugins.o core/lock.o core/cache.o core/daemons.o core/errors.o core/hash.o core/master_events.o core/chunked.o core/queue.o core/event.o core/signal.o core/strings.o core/progress.o core/timebomb.o core/ini.o core/fsmon.o core/mount.o core/metrics.o core/plugins_builder.o core/sharedarea.o core/rpc.o core/gateway.o core/loop.o core/cookie.o core/querystring.o core/rb_timers.o core/transformations.o core/uwsgi.o proto/base.o proto/uwsgi.o proto/http.o proto/fastcgi.o proto/scgi.o proto/puwsgi.o lib/linux_ns.o core/zlib.o core/yaml.o core/dot_h.o core/config_py.o plugins/python/python_plugin.o plugins/python/pyutils.o plugins/python/pyloader.o plugins/python/wsgi_handlers.o plugins/python/wsgi_headers.o plugins/python/wsgi_subhandler.o plugins/python/web3_subhandler.o plugins/python/pump_subhandler.o plugins/python/gil.o plugins/python/uwsgi_pymodule.o plugins/python/profiler.o plugins/python/symimporter.o plugins/python/tracebacker.o plugins/python/raw.o plugins/gevent/gevent.o plugins/gevent/hooks.o plugins/ping/ping_plugin.o plugins/cache/cache.o plugins/nagios/nagios.o plugins/rrdtool/rrdtool.o plugins/carbon/carbon.o plugins/rpc/rpc_plugin.o plugins/corerouter/cr_common.o plugins/corerouter/cr_map.o plugins/corerouter/corerouter.o plugins/fastrouter/fastrouter.o plugins/http/http.o plugins/http/keepalive.o plugins/http/https.o plugins/http/spdy3.o plugins/ugreen/ugreen.o plugins/signal/signal_plugin.o plugins/syslog/syslog_plugin.o plugins/rsyslog/rsyslog_plugin.o plugins/logsocket/logsocket_plugin.o plugins/router_uwsgi/router_uwsgi.o plugins/router_redirect/router_redirect.o plugins/router_basicauth/router_basicauth.o plugins/zergpool/zergpool.o plugins/redislog/redislog_plugin.o plugins/mongodblog/mongodblog_plugin.o plugins/router_rewrite/router_rewrite.o plugins/router_http/router_http.o plugins/logfile/logfile.o plugins/router_cache/router_cache.o plugins/rawrouter/rawrouter.o plugins/router_static/router_static.o plugins/sslrouter/sslrouter.o plugins/spooler/spooler_plugin.o plugins/cheaper_busyness/cheaper_busyness.o plugins/symcall/symcall_plugin.o plugins/transformation_tofile/tofile.o plugins/transformation_gzip/gzip.o plugins/transformation_chunked/chunked.o plugins/transformation_offload/offload.o plugins/router_memcached/router_memcached.o plugins/router_redis/router_redis.o plugins/router_hash/router_hash.o plugins/router_expires/expires.o plugins/router_metrics/plugin.o plugins/transformation_template/tt.o plugins/stats_pusher_socket/plugin.o -lpthread -lm -rdynamic -ldl -lz -lpthread -ldl -lutil -lm -lpython3.7m -lcrypt
/usr/bin/ld: cannot open output file /usr/local/bin/uwsgi: Permission denied
collect2: error: ld returned 1 exit status

----------------------------------------
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-wg1rqgn8/uwsgi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-r99m_baa-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-wg1rqgn8/uwsgi/


What’s amazing to me is how common this is with Python. It really is amazing how broken the dependencies are. In the world of Ruby, I can use gem and bundler. Once in a while I have a problem with something like nokogiri, but Ruby Bundler mostly works. And in the world of Clojure, I’ve never had a problem, because Leiningen is best build tool ever created.

But in Python, every install has a problem, always.

I won’t even talk about weirdness like this:

/usr/bin/ld: cannot open output file /usr/local/bin/uwsgi: Permission denied

Why any software output to /usr/local/bin/ ? Does that sound like an appropriate location for output?

So anyway, in Python, the most common reason things are broken is that Python3 is a completely separate language versus Python2, so I try:

pip3 install uwsgi

This gets me the same error.

I go looking for answers and I find:

A curious fact: when I ask my co-workers “Why has Python gotten so popular over the last 10 years?” one of the main answers I’m given is “It’s really easy for beginners to get started”. And yet, for me, when I was a beginner, one of the main things I wanted was the ability to write code without having to dig deep into the internals of the system that I was using. Back in 2001, when I wrote PHP code, I could write all day without having to think too much about the way PHP connected to port 80 on the server. But in Python I have to think about it, and the dependency management system is broken, so I can’t just install the libraries I need.

I seriously do not get why the tech industry is sticking with Python. And not just sticking with it, but doubling down on it. As near as I can tell, Python is rising in popularity, even as PHP and Ruby die away.

“Easy for beginners” was Hypercard, and I loved writing stacks back in 1994, and the C extensions were easy to install. And 24 years later have passed, and “Easy for beginners” seems to have gotten much more difficult. How is that even possible? Why is the tech industry going in the wrong direction?

=================

[ [ UPDATE 2018-08-11 ] ]

./manage.py runserver 0.0.0.0:8000

Which gives me:

https://stackoverflow.com/questions/42611593/how-to-solve-syntaxerror-on-autogenerated-manage-py

The top 2 suggestions amount to “make sure you are running Django using the correct version of Python” so I try this:

There is no other language eco-system that is so utterly broken in the year 2018. I know I’ve complained, on this blog, about the times when Ruby Bundler can’t handle compile Nokogiri, but that seems minor compared to the version conflicts in Python, which are pervasive.

Oh, wait, I see, maybe I just need to reinstall Django. Okay, let’s do that:

It is just amazing how pervasive the version conflicts are. They appear on almost every command that I run. Compare this to PHP or Ruby or Javascript or Java or Scala or Clojure or literally any other language. There is simply no other popular technology that has such pervasive version conflicts.

Source

Check out my book:

August 11, 2018 8:10 pm

"I should add, I feel that Python 1.x and 2.x added something to the world of technology. Early Python was both..."

August 11, 2018 10:44 am

"Ryan, about this: " the default should be to not try to install packages system-wide". Agreed. I think that is..."

August 10, 2018 8:23 pm

"I suspect your problem is that you missed the step in the instructions where it told you to run all those comm..."

August 5, 2018 4:41 pm

"This one is very intense reading thank you..."

July 27, 2018 12:04 am

"Many coed leagues require a certain %age of women players, like volleyball 2 out of 5, in a lot of leagues. ..."

July 15, 2018 3:17 pm

"Good to know that I wasn't too late :)..."

July 15, 2018 8:32 am

"Thank you for these comments regarding Anki. It sounds useful. I can imagine having the full context of the se..."

July 15, 2018 6:32 am

"Anki limits how much you review. I have 10000+ cards across my decks but the program arranges them in such a w..."

July 6, 2018 3:42 pm

"RK, your comment is irresponsible and pure FUD. These technologies are not being deprecated. Some of them have..."

July 5, 2018 10:21 am

"Deprecation warning: The Packer, Artifact Registry and Terraform Enterprise (Legacy) features of Atlas will no..."

June 25, 2018 11:58 am

"Solid read. I will say that it seems like there is a lot of momentum growing around Ansible these days. It see..."

June 21, 2018 11:25 pm

"Thank you for this. Sometimes I feel like I'm living in a bizarre, pro-Docker for everything world. I am prima..."

May 30, 2018 4:35 pm

"Thank you for this beautifully written and relevant piece. I think it has saved me a whole world of pain. ..."

May 29, 2018 10:46 am

"Andrew Reilly, thank you for writing. You make several good points, especially this: "Also an extra layer o..."

May 28, 2018 11:17 pm

"Nice article, thanks! I think that it's interesting to also look around at the problem from a greater dista..."

August 10, 2018
8:23 pm

By Ryan

I suspect your problem is that you missed the step in the instructions where it told you to run all those commands in a virtualenv.

Alternatively you can run ‘pip install’ with the ‘–user’ flag to install into your users home directory instead of system-wide. Or run ‘pip install’ as a user that has permission to write to /use/local/bin.

I agree though that the defaults are crappy and the default should be to not try to install packages system-wide…

August 11, 2018
10:44 am

By lawrence

Ryan, about this: ” the default should be to not try to install packages system-wide”. Agreed. I think that is one of those things that have driven the Python community towards Docker, because in Docker it is safer to have everything global. But that doesn’t strike me as a high productivity strategy in 2018. See “Docker protects a programming paradigm that we should get rid of”