August 10th, 2018

Why does anyone use Python?

(written by lawrence krubner, however indented passages are often quotes). You can contact lawrence at: lawrence@krubner.com, or follow me on Twitter.

Summary: Python 1.x and 2.x brought some interesting ideas to the tech world, but the upgrade to 3.x has brought an astonishing number of problems in terms of version conflicts and dependency management. Python no longer offers ease of setup, as any non-trivial system will run into version problems. If you are a CTO, you should consider using a technology stack that has a long history of being stable, such as any of the languages that run on the JVM and compile to Java bytecode.

Over the last 20 years I’ve written in software in PHP and Ruby and Javascript and Java and Swift and Scala and Perl and Coffeescript and TypeScript and Clojure and Python. My favorite is Clojure. Python is competitive in the race for “most annoying eco-system.”

Despite the fact that PHP has some hilarious essays about how broken it is I used to enjoy writing PHP code, especially PHP 4, before they added unnecessary Object Oriented junk in PHP 5. I am happy to praise PHP for having a dependency management model that mostly works.

But Python? It seems like everything is broken in Python all the time. I’ve written about this a few times lately, and I’m just struck, once again, tonight, how the simplest and most obvious things send me spiraling into lost hours, trying to figure out why some command isn’t working.

So for instance, tonight, the intern who is working for us for the summer said “Can you help me connect my Django app to the web?” and I said “sure”. I looked around for a tutorial about attaching Django to Nginx, and I found this:

Which tells me to do this:

pip install uwsgi

and I get:

pip install uwsgi
Collecting uwsgi
Using cached https://files.pythonhosted.org/packages/a2/c9/a2d5737f63cd9df4317a4acc15d1ddf4952e28398601d8d7d706c16381e0/uwsgi-2.0.17.1.tar.gz
Installing collected packages: uwsgi
Running setup.py install for uwsgi ... error
Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-wg1rqgn8/uwsgi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-r99m_baa-record/install-record.txt --single-version-externally-managed --compile:
/usr/lib64/python3.7/distutils/dist.py:274: UserWarning: Unknown distribution option: 'descriptions'
warnings.warn(msg)
running install
using profile: buildconf/default.ini
detected include path: ['/usr/lib/gcc/x86_64-redhat-linux/7/include', '/usr/local/include', '/usr/include']
Patching "bin_name" to properly install_scripts dir
detected CPU cores: 2
configured CFLAGS: -O2 -I. -Wall -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -DUWSGI_HAS_IFADDRS -DUWSGI_ZLIB -DUWSGI_LOCK_USE_MUTEX -DUWSGI_EVENT_USE_EPOLL -DUWSGI_EVENT_TIMER_USE_TIMERFD -DUWSGI_EVENT_FILEMONITOR_USE_INOTIFY -DUWSGI_VERSION="\"2.0.17.1\"" -DUWSGI_VERSION_BASE="2" -DUWSGI_VERSION_MAJOR="0" -DUWSGI_VERSION_MINOR="17" -DUWSGI_VERSION_REVISION="1" -DUWSGI_VERSION_CUSTOM="\"\"" -DUWSGI_YAML -DUWSGI_PLUGIN_DIR="\".\"" -DUWSGI_DECLARE_EMBEDDED_PLUGINS="UDEP(python);UDEP(gevent);UDEP(ping);UDEP(cache);UDEP(nagios);UDEP(rrdtool);UDEP(carbon);UDEP(rpc);UDEP(corerouter);UDEP(fastrouter);UDEP(http);UDEP(ugreen);UDEP(signal);UDEP(syslog);UDEP(rsyslog);UDEP(logsocket);UDEP(router_uwsgi);UDEP(router_redirect);UDEP(router_basicauth);UDEP(zergpool);UDEP(redislog);UDEP(mongodblog);UDEP(router_rewrite);UDEP(router_http);UDEP(logfile);UDEP(router_cache);UDEP(rawrouter);UDEP(router_static);UDEP(sslrouter);UDEP(spooler);UDEP(cheaper_busyness);UDEP(symcall);UDEP(transformation_tofile);UDEP(transformation_gzip);UDEP(transformation_chunked);UDEP(transformation_offload);UDEP(router_memcached);UDEP(router_redis);UDEP(router_hash);UDEP(router_expires);UDEP(router_metrics);UDEP(transformation_template);UDEP(stats_pusher_socket);" -DUWSGI_LOAD_EMBEDDED_PLUGINS="ULEP(python);ULEP(gevent);ULEP(ping);ULEP(cache);ULEP(nagios);ULEP(rrdtool);ULEP(carbon);ULEP(rpc);ULEP(corerouter);ULEP(fastrouter);ULEP(http);ULEP(ugreen);ULEP(signal);ULEP(syslog);ULEP(rsyslog);ULEP(logsocket);ULEP(router_uwsgi);ULEP(router_redirect);ULEP(router_basicauth);ULEP(zergpool);ULEP(redislog);ULEP(mongodblog);ULEP(router_rewrite);ULEP(router_http);ULEP(logfile);ULEP(router_cache);ULEP(rawrouter);ULEP(router_static);ULEP(sslrouter);ULEP(spooler);ULEP(cheaper_busyness);ULEP(symcall);ULEP(transformation_tofile);ULEP(transformation_gzip);ULEP(transformation_chunked);ULEP(transformation_offload);ULEP(router_memcached);ULEP(router_redis);ULEP(router_hash);ULEP(router_expires);ULEP(router_metrics);ULEP(transformation_template);ULEP(stats_pusher_socket);"
*** uWSGI compiling server core ***
*** uWSGI compiling embedded plugins ***
gcc -pthread -o /usr/local/bin/uwsgi -L/usr/lib64 -Wl,-rpath,/usr/lib64 core/utils.o core/protocol.o core/socket.o core/logging.o core/master.o core/master_utils.o core/emperor.o core/notify.o core/mule.o core/subscription.o core/stats.o core/sendfile.o core/async.o core/master_checks.o core/fifo.o core/offload.o core/io.o core/static.o core/websockets.o core/spooler.o core/snmp.o core/exceptions.o core/config.o core/setup_utils.o core/clock.o core/init.o core/buffer.o core/reader.o core/writer.o core/alarm.o core/cron.o core/hooks.o core/plugins.o core/lock.o core/cache.o core/daemons.o core/errors.o core/hash.o core/master_events.o core/chunked.o core/queue.o core/event.o core/signal.o core/strings.o core/progress.o core/timebomb.o core/ini.o core/fsmon.o core/mount.o core/metrics.o core/plugins_builder.o core/sharedarea.o core/rpc.o core/gateway.o core/loop.o core/cookie.o core/querystring.o core/rb_timers.o core/transformations.o core/uwsgi.o proto/base.o proto/uwsgi.o proto/http.o proto/fastcgi.o proto/scgi.o proto/puwsgi.o lib/linux_ns.o core/zlib.o core/yaml.o core/dot_h.o core/config_py.o plugins/python/python_plugin.o plugins/python/pyutils.o plugins/python/pyloader.o plugins/python/wsgi_handlers.o plugins/python/wsgi_headers.o plugins/python/wsgi_subhandler.o plugins/python/web3_subhandler.o plugins/python/pump_subhandler.o plugins/python/gil.o plugins/python/uwsgi_pymodule.o plugins/python/profiler.o plugins/python/symimporter.o plugins/python/tracebacker.o plugins/python/raw.o plugins/gevent/gevent.o plugins/gevent/hooks.o plugins/ping/ping_plugin.o plugins/cache/cache.o plugins/nagios/nagios.o plugins/rrdtool/rrdtool.o plugins/carbon/carbon.o plugins/rpc/rpc_plugin.o plugins/corerouter/cr_common.o plugins/corerouter/cr_map.o plugins/corerouter/corerouter.o plugins/fastrouter/fastrouter.o plugins/http/http.o plugins/http/keepalive.o plugins/http/https.o plugins/http/spdy3.o plugins/ugreen/ugreen.o plugins/signal/signal_plugin.o plugins/syslog/syslog_plugin.o plugins/rsyslog/rsyslog_plugin.o plugins/logsocket/logsocket_plugin.o plugins/router_uwsgi/router_uwsgi.o plugins/router_redirect/router_redirect.o plugins/router_basicauth/router_basicauth.o plugins/zergpool/zergpool.o plugins/redislog/redislog_plugin.o plugins/mongodblog/mongodblog_plugin.o plugins/router_rewrite/router_rewrite.o plugins/router_http/router_http.o plugins/logfile/logfile.o plugins/router_cache/router_cache.o plugins/rawrouter/rawrouter.o plugins/router_static/router_static.o plugins/sslrouter/sslrouter.o plugins/spooler/spooler_plugin.o plugins/cheaper_busyness/cheaper_busyness.o plugins/symcall/symcall_plugin.o plugins/transformation_tofile/tofile.o plugins/transformation_gzip/gzip.o plugins/transformation_chunked/chunked.o plugins/transformation_offload/offload.o plugins/router_memcached/router_memcached.o plugins/router_redis/router_redis.o plugins/router_hash/router_hash.o plugins/router_expires/expires.o plugins/router_metrics/plugin.o plugins/transformation_template/tt.o plugins/stats_pusher_socket/plugin.o -lpthread -lm -rdynamic -ldl -lz -lpthread -ldl -lutil -lm -lpython3.7m -lcrypt
/usr/bin/ld: cannot open output file /usr/local/bin/uwsgi: Permission denied
collect2: error: ld returned 1 exit status

----------------------------------------
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-wg1rqgn8/uwsgi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-r99m_baa-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-wg1rqgn8/uwsgi/


—————
—————
—————

What’s amazing to me is how common this is with Python. In the world of Ruby, I can use gem and bundler. Once in a while I have a problem with something like nokogiri, but Ruby Bundler mostly works. And in the world of Clojure, I’ve never had a problem, because Leiningen is best build tool ever created.

So anyway, in Python, the most common reason things are broken is that Python3 is a completely separate language versus Python2, so I try:

pip3 install uwsgi

This gets me the same error.

I go looking for answers and I find:

A curious fact: when I ask my co-workers “Why has Python gotten so popular over the last 10 years?” one of the main answers I’m given is “It’s really easy for beginners to get started”. And yet, for me, when I was a beginner, one of the main things I wanted was the ability to write code without having to dig deep into the internals of the system that I was using. Back in 2001, when I wrote PHP code, I could write all day without having to think too much about the way PHP connected to port 80 on the server. But in Python I have to think about it, and the dependency management system is broken, so I can’t just install the libraries I need.

I seriously do not get why the tech industry is sticking with Python. And not just sticking with it, but doubling down on it. As near as I can tell, Python is rising in popularity, even as PHP and Ruby die away.

“Easy for beginners” was Hypercard, and I loved writing stacks back in 1994, and the C extensions were easy to install. And 24 years later have passed, and “Easy for beginners” seems to have gotten much more difficult. How is that even possible? Why is the tech industry going in the wrong direction?

=================

[ [ UPDATE 2018-08-11 ] ]

./manage.py runserver 0.0.0.0:8000

Which gives me:

https://stackoverflow.com/questions/42611593/how-to-solve-syntaxerror-on-autogenerated-manage-py

The top 2 suggestions amount to “make sure you are running Django using the correct version of Python” so I try this:

Oh, wait, I see, maybe I just need to reinstall Django. Okay, let’s do that:

Compare this to PHP or Ruby or Javascript or Java or Scala or Clojure or literally any other language. There is simply no other popular technology that has such pervasive version conflicts.

I know I’ve complained, on this blog, about the times when Ruby Bundler can’t compile Nokogiri, but that seems minor compared to the version conflicts in Python.

————————————

————————————

[ [ UPDATE 2018-08-19 ] ]

More struggles with bizarre version conflicts. This:

/usr/local/lib/python3.7/site-packages/gunicorn/gunicorn ecommerce.wsgi:application –bind 0.0.0.0:8000

gives me:

ImportError: cannot import name ‘cookies’ from ‘http’ (/usr/local/lib/python3.7/site-packages/gunicorn/http/__init__.py)

Hmm, maybe this is a version conflict, so let’s upgrade everything.

I feel badly that I ever complained about dependency management in the world of Ruby. Whatever problems Ruby has, they seem like a dream compared to Python. I’ve really never worked with an eco-system where the dependency management is as broken as what I face in Python.

And there is also the fact that the number of conflicts are nearly infinite, and the systems seems creative in finding new combinations, so I don’t seem to be able to find my errors on Google.

——-
——-
——-

UPDATE 2018-08-21

Starting over again on a fresh EC2 instance, trying to do everything the “correct” way. Interesting that no matter what I do, there is always some error:

This leaves me with the impression that no one in the Python community is certain how to resolve all of these version conflicts.

——————–

——————–

——————–

Another observation about the culture of the community of Python programmers:

Most of the blog posts, tutorials, and StackOverflow posts suggest that one has to activate a virtual environment before installing something like gunicorn, but the official documentation creates the misleading illusion that installation is easy:

Of course, in the real world, you would never just do what you see in this screenshot. There would be several other steps.

Source

Check out my books:

August 21, 2019 8:47 am

"Hi there and I agree completely but I can resume as follow: Docker promises simplicity, i.e. IT IS EASY. ..."

August 20, 2019 2:29 pm

"Which is fine. Like I said, there are dance scenes that have strict “no alcohol” rules. That might appeal to y..."

August 20, 2019 1:35 pm

""Promise of an early bed" - the whiff of danger keeps me away from many venues like the one you describe...."

August 20, 2019 12:22 am

"I think any time you go to any club there is the possibility of running into an angry person, maybe a person w..."

August 19, 2019 7:56 pm

"I'm confused. You and your friends went out, had a fight, and it's still a great place to go to? Maybe..."

August 18, 2019 8:57 pm

"You seem to have little patience for people who choose different tech paths than you. Although it looks like o..."

August 18, 2019 8:53 pm

"You don't suffer slights well, do you? Others who choose to waste time with dumb tech, do they keep you up at ..."

August 18, 2019 8:34 pm

"Me again. I've worked for a company that focuses on containerized applications for some time now. There is abs..."

August 18, 2019 8:00 pm

"To build on my last statement, I'm not trying to show that I'm "smarter." I'm probably not, or if I am, who gi..."

August 18, 2019 7:48 pm

"You think that containerization is going anywhere? I agree that it isn't strictly necessary, but you mistake y..."

August 18, 2019 7:40 pm

"If you weren't criticizing Docker, why did your frame the article as anti the Docker/Kubernetes/Python troika?..."

August 13, 2019 1:34 pm

"No tool is every perfect, and you're certainly free to use whatever you want. But, I doubt very much the huge ..."

August 5, 2019 5:23 pm

"DangerNorm, as to your last point, I was recently consulting with a startup that focused on the privacy of med..."

August 5, 2019 3:55 pm

"This is tangential to the above point, but on the subject of mental health, I'll also take this chance to say ..."

August 5, 2019 3:51 pm

"Your description of most people's response is correct, but I consider it to be a problem, rather than the corr..."

August 10, 2018
8:23 pm

By Ryan

I suspect your problem is that you missed the step in the instructions where it told you to run all those commands in a virtualenv.

Alternatively you can run ‘pip install’ with the ‘–user’ flag to install into your users home directory instead of system-wide. Or run ‘pip install’ as a user that has permission to write to /use/local/bin.

I agree though that the defaults are crappy and the default should be to not try to install packages system-wide…

August 11, 2018
10:44 am

By lawrence

Ryan, about this: ” the default should be to not try to install packages system-wide”. Agreed. I think that is one of those things that have driven the Python community towards Docker, because in Docker it is safer to have everything global. But that doesn’t strike me as a high productivity strategy in 2018. See “Docker protects a programming paradigm that we should get rid of”