Shanghai Building to be Demolished

Technology

November 24th, 2022

In Technology

No Comments

Why is 99% reliability a problem for Twitter?

I’m concerned that non-technical people might think that 99% reliability is very good, so I’m writing this to explain the real situation. This is relevant to any website or software, but I’m going to focus on Twitter.

I’ve been at early stage startups where we were stretched very thin, and we had to live with 99% reliability. We were embarrassed about it, but we could explain to our potential customers “we are very early stage” and they would accept that, at ...

Read More Source

May 29th, 2022

In Technology

No Comments

Don’t use Airflow

The comments on this thread are fairly brutal towards Airflow:

 I believe airflow was probably the right choice at the time we started. We were a small team, and deploying airflow was a major shortcut that more or less handled orchestration so we could focus on other problems. With the aid of hindsight, we would have been better off spinning off our own scheduler some time in the first year of the project. Like I mentioned in my OP, we have ...

Read More Source

May 29th, 2022

In Technology

No Comments

Everyone hates NFS yet everyone uses NFS

For more than 10 years I’ve heard people tell me that NFS is obsolete and no sensible engineer would ever use it. Nevertheless, it remains useful and so engineers use it:

After some experimentation we found that we could vastly improve performance across our Airflow environments by running an NFS (network file system) server within the Kubernetes cluster. We then mounted this NFS server as a read-write-many volume into the worker and scheduler pods. We wrote a custom script which synchronizes ...

Read More Source

May 27th, 2022

In Technology

No Comments

What if something changes?

This is both funny and true:

“What if it changes?” isn’t just a question. It’s a powerful heuristic for software design that can be used to justify almost anything. Everyone should use it more. It’s great precisely because it’s rooted in pure speculation. Once you’ve freed yourself from the baggage of reality, there’s nothing easier than inventing scenarios where your special code will be useful under the new imaginary future conditions. If you encounter any pushback against your defensive layer cake of abstraction, ...

Read More Source

May 27th, 2022

In Technology

No Comments

Software is making hospitals worse — a funny story

(I wrote this as an email for my friends, but then I decided to also share this with the public.)

Okay, I already mentioned that my mom was at the hospital for a week, with an infected foot, and she got lots of antibiotics, and they sent her home on Monday?

Every day I went over there for 6 hours and tried to entertain her a bit. It’s easy to go crazy in a hospital, as it is so very boring. My ...

Read More Source

May 26th, 2022

In Technology

No Comments

The simplicity of skip list is the source of its power

This is a very interesting post. I wish Single Store was open source.

3) Lock-Free

A lock-free or non-blocking algorithm is one in which some thread is always able to make progress, no matter how all the threads’ executions are interleaved by the OS. SingleStore is designed to support highly concurrent workloads running on hardware with many cores. These goals makes lock-free algorithms desirable for SingleStore. (See our original blog post on lock-free algorithms and our description of sync replication, by Nate Horan.)The algorithms for ...

Read More Source

May 25th, 2022

In Technology

No Comments

The cost of no generics in a programming language: no way to inline sort functions

Interesting post:

The idea is that the compiler is able to emit code specifically for the invocation we use. Instead of having to emit a function call on each invocation, the compare call will usually be inlined and the cost of invocation is completely eliminated.

Java is the only one on this list that has a different approach. Instead of using generics at compile time, it is actually doing a dispatch of the code to optimized routines based on runtime types. That ...

Read More Source

April 17th, 2022

In Technology

No Comments

Why do some companies use a database that is not a relational RDBMS?

I saw this question on Reddit. This would be my answer:

I’ve worked at many companies where they used a RDBMS but they cheated massively, so in a sense it is a bit more honest to switch to something else.

For instance, when I was at PrivCo.com, they were using MySQL, but they had many tables that they were using as a cache. Much worse, they had no clear sense of which of their tables were truly canonical. Inside the tables that ...

Read More Source

April 3rd, 2022

In Technology

No Comments

I’ve never written a file with 11,000 lines of code, but I have often built Clojure projects like this, with everything in one file. I think I might have once had a file with 4,000 lines of code. Maybe 5,000? A complete system might be 5 apps, working together, each made of one large file. It does help with some things. Especially if I try to on-board another programmer, if they don’t know Clojure very well, using one file means ...

Read More Source

March 23rd, 2022

In Technology

No Comments

The specialization that comes with growth

Lethain’s experience at Uber. This is very good and worth reading:

When I joined Uber, the Infrastructure organization consisted of three teams (whose names were unhelpfully generic, so I’m renaming them a bit for clarity): developer productivity who worked on build and test (~4 engineers), storage engineering (~6 engineers) who worked on scaling real-time storage, and operations (~5 engineers) who did everything else to support the company’s ~200 engineers, ~2,000 employees, and ~400% YoY growth in both usage and engineering ...

Read More Source

August 31st, 2021

In Technology

No Comments

Software development theories are collective fictions

I like this essay very much. The only part I disagree with is that software development amounts to a collective fiction, like religion, which represents a kind of behavior can be traced back to prehistoric times. Rather, I see the problem as radically modern: the desire to take an art and make it a science. An unrelated example would be the way universities, over the last 70 years, have opened many “Department of Political Science” and yet no university has ...

Read More Source

August 17th, 2021

In Technology

No Comments

An unstudied problem: query optimization for streaming data

Interesting:

I haven’t done any kind of systematic literature review recently, so the directions below are kind of a random sampling of things I happen to be have stumbled over.

Exploring the plan space seems like a first step. It seems likely to behave differently than in a batch setting. For example, in a batch query when you’re joining a big relation against a small relation, you want to build an index on the small relation and scan the big relation against it, ...

Read More Source

August 16th, 2021

In Technology

No Comments

The Lisp Curse by Rudolf Winestock

[Note from Lawrence Krubner: I worry about linkrot, and I worry about the disappearance of those essays that had an impact on my intellectual growth. How can I talk about certain ideas in the future if I can’t point people to the essays that shaped my thinking? For some reason, Rudolf Winestock’s weblog is offline. I’m hoping Rudolf Winestock will soon restore their blog. Until then, I’m going to temporarily host their essay, the Lisp Curse, here, as it is ...

Read More Source

September 30th, 2019

In Technology

11 Comments

My final post regarding the flaws of Docker / Kubernetes and their eco-system

With a surprising number of Docker advocates, they advocate for Docker/Kubernetes in a vacuum, as if the question was “Do you want devops, or not?” when the correct question is “Of this set of dozens of devop technologies, which subset is the right one for my company?” I’m often subjected to a three part argument that’s missing its middle part:

1.) Do you want easy development, deployment, consistency, networking, security, isolation and infinite scalability?

2.) ???

3.) In conclusion, Docker!!!!!!

Some of the conversations ...

Read More Source

September 23rd, 2019

In Technology

No Comments

Amazon Mechanical Turk workers demonstrated racial bias

Interesting:

Created in 2009 by researchers at Princeton and Stanford, the online image database has been widely used by machine learning projects. The program has pulled more than 14 million images from across the web, which have been categorized by Amazon Mechanical Turk workers — a crowdsourcing platform through which people can earn money performing small tasks for third parties. According to the results of an online project by AI researcher Kate Crawford and artist Trevor Paglen, prejudices in that labor ...

Read More Source

September 22nd, 2019

In Technology

No Comments

Every deploy, after all, is a unique and never-to-be-replicated combination of artifact, environment, infra, and time of day

Interesting:

And if testing is about uncertainty, you “test” any time you deploy to production. Every deploy, after all, is a unique and never-to-be-replicated combination of artifact, environment, infra, and time of day. By the time you’ve tested, it has changed.

Once you deploy, you aren’t testing code anymore, you’re testing systems—complex systems made up of users, code, environment, infrastructure, and a point in time. These systems have unpredictable interactions, lack any sane ordering, and develop emergent properties which perpetually and eternally ...

Read More Source

September 22nd, 2019

In Technology

No Comments

The problems of stateful data in Kubernetes

Interesting:

StatefulSet for deploying stateful Pods

StatefulSet is the abstraction that was supposed to solve all these issues. They give each Pod in the set a unique identifier, and the StatefulSet documentation says the following about them and intended use:

StatefulSets are valuable for applications that require one or more of the following.

* Stable, unique network identifiers.

* Stable, persistent storage.

* Ordered, graceful deployment and scaling.

* Ordered, graceful deletion and termination.

* Ordered, automated rolling updates.

Now we’re talking! Stable network and storage! Even ordering in ...

Read More Source

September 22nd, 2019

In Technology

No Comments

Making a snapshot of etcd to backup Kubernetes

Interesting, but I worry about the complexity of what is going on here:

etcd snapshot explanation

The second command needs a bit more explaining. First of all, the idea is to create a snapshot of the etcd database. This is done by communicating with the running etcd instance in Kubernetes and asking it to create a snapshot. The reason for the very long command is basically to avoid messing with etcd running in Kubernetes as much as possible. We are launching ...

Read More Source

September 9th, 2019

In Technology

No Comments

We all test in production

All of this is good. I touched on a small piece of these issues when I wrote “How ignorant am I, and how do I formally specify that in my code?”

Also see “High Availability is not compatible with a MVP“.

Interesting:

When we say “production,” we typically mean the constellation of all these things and more. Despite our best efforts to abstract away such pesky low-level details as the firmware version on your eth0 card on an EC2 instance, I’m here to ...

Read More Source

September 8th, 2019

In Technology

No Comments

What if the operating system was a database?

This idea has been kicked around for decades. I wonder why so little progress is made?

The configuration of a UNIX system is specified and controlled by a huge tangle of plain text files: /etc/hosts, sendmail.cf, syslog.conf, inetd.conf, /etc/uucp/Systems to name just very few. .INI files on some other systems are also plain ASCII. Even MacOS caved in a little with a System Folder:Hosts, although it is a very isolated example on a Mac. Note that just because symbols displayed on ...

Read More Source

September 1st, 2019

In Technology

No Comments

A supportive community brings Clojure to the world

At some point I hope to write more about the Clojure community, for now, I link to this:

Over the last couple of years, the Clojure community has become an invaluable part of my life. It’s a space where all of me has been welcomed and accepted, since day one. The kindness, support and inspiration I have received from people in the community is boundless. My heart has been touched many times by the deep and genuine connections, old ones keep ...

Read More Source

August 31st, 2019

In Technology

No Comments

Lisp is so powerful that problems which are technical issues in other programming languages are social issues in Lisp

Interesting:

Consider the case of Scheme, again. Since making Scheme object-oriented is so easy, many Scheme hackers have done so. More to the point, many individual Scheme hackers have done so. In the 1990s, this led to a veritable warehouse inventory list of object-oriented packages for the language. The Paradox of Choice, alone, guaranteed that none of them would become standard. Now that some Scheme implementations have their own object orientation facilities, it’s not so bad. Nevertheless, the fact that many ...

Read More Source

August 26th, 2019

In Technology

2 Comments

Don’t waste $1 million on devops infrastructure that you’ll never need

Samy Dindane asked me a question:

How would you deploy a simple React/NodeJS app?

I responded on Twitter:

Deployment is a big question but I’ll try to summarize. First, ask yourself, do you need to High Availability? When I ask CEOs of early stage startups, they always say “Yes” yet I think only 10% really need High Availability.

I think of High Availability as implying multiple servers, multiple load balancers, multiple failover databases, multiple data centers, multiple regions. Lots of redundancy, probably configured for ...

Read More Source

August 12th, 2019

In Technology

No Comments

How Homakov hacked Github

Apparently I never linked to this? I was impressed with this. A few minor hacks that Github already knew about, and had looked at, and had decided they weren’t important. But Homakov found a way to link them together into a serious attack.

Post external references 1 Source

August 8th, 2019

In Technology

No Comments

Please don’t touch the keyboard

Come on. This technique can be improved.

Source

July 18th, 2019

In Technology

No Comments

Ethereum is dangerous and Pact offers a much better blockchain smart contract language

What strikes me is how carefully they’ve thought about the problems with Ethereum, and other blockchain technologies. They deliberately make sure that Pact is NOT Turing complete, so they can avoid some of the hacks that took place with Ethereum. They emphasize a declarative style. And yet they also make it so the contracts can be fairly upgraded.

This is from the article:

However, the added versatility offered by Solidity smart contracts comes at a great cost in safety and complexity, jeopardizing ...

Read More Source

July 4th, 2019

In Technology

No Comments

The right way to do Agile

The best implementation of Agile that I’ve seen was when I was at ShermansTravel.com and Mark Herschberg was CTO. The system went like this:

Herschberg reckoned that each developer could get 5 hours of real work done each day (the rest of the day got eaten up by email, conversations, research, and other incidentals). We had a crew of 5 programmers, so we had 125 hours a week of real programming. The sprints were 2 weeks long, so we had 250 ...

Read More Source

June 17th, 2019

In Technology

No Comments

A real life problem with our Docker use that cost us 3 days of lost work

Sadly, no one saw this because I originally posted it as an update to “Docker protects a programming paradigm that we should get rid of” but just now, on Hacker News, steve_taylor wrote:

docker-compose up -d

What’s not easy about that?

My response? This is the sales pitch for Docker:

docker-compose up -d

But this Slack conversation is the reality (we lost 3 days before we were able to fix the issue, a complex interplay of the Docker cache and the way the Dockerfile ...

Read More Source

June 15th, 2019

In Technology

No Comments

How does your frontend know the id of a document if you’re using an async architecture? Rely on UUIDs.

Building an async system is more complicated than building a sequential system, but it can offer competitive advantages. For instance, you can build mobile apps that work offline, and later sync their data with your system, and this ability to function to work offline can enable uses that your competitors don’t have. This is what allowed Handshake CRM to thrive, even though they started after the CRM market had seemingly consolidated down to a handful of winners. Given the dominance ...

Read More Source

June 15th, 2019

In Technology

No Comments

Caches are cheap, build a lot of them for the frontend

There was a stretch, maybe from 2005 to 2015, which will be remembered as the era of simple Ajax uses. The era started when Sam Stephenson released the Prototype library, but it really got going once jQuery was released. It was an era of Javascript fetching data via Ajax, and relying on the backend code to decide what data is sent in response to each request.

That era is evolving into something different. Frontends are now complicated enough that the ...

Read More Source

June 15th, 2019

In Technology

No Comments

Why are software developers confused by Kafka and immutable logs?

Simplicity. Why is it so hard to keep things simple? There are many concepts that we use in software which arise from common sense which we then encumber with needless complexity. Why?

For instance, logs. It’s great to have a record of things. If you’ve built some software that gathers information, it is often very useful to know the order in which information arrived in your system. That is, it is insightful to know the history of events that built ...

Read More Source

May 30th, 2019

In Technology

No Comments

The collision of capital and idealism over a technology

This is a good article, but is it appropriate to say that a technology is simply going through some growing pains when the technology is 25 years old? Doesn’t the long history suggest the problem here is a complex one, with a long history of its own? Python was a dying language until Google decided to make large investments in it. Now it is the main language for Data Science. This is a language that started off with a certain ...

Read More Source

May 30th, 2019

In Technology

5 Comments

Billions were wasted on Hadoop startups, and the same will eventually be true of Docker

The hype over Hadoop has died away and many of the investments now seem to have been terrible investments:

MapR Technologies Inc., one of the troika of unicorn startups that emerged from the early days of the big-data movement, may cut up to 122 jobs and shut down its Santa Clara, California headquarters if it can’t secure additional funding.

The company, which raised a total of $280 million in financing since it was founded in 2009 and whose market capitalization once ...

Read More Source

May 29th, 2019

In Technology

No Comments

Could client side Java make a comeback?

Interesting:

Gluon has always been very active in the Java Mobile Enterprise area. One of the big benefits of Java on the mobile and embedded client, is the fact that 12 million Java developers can use their skills to create mobile and embedded applications. In the past, the runtime for mobile platforms always lagged behind the runtime for desktop.

When we announced GluonVM, we said we would aim to bring Java on mobile at the same level as Java on desktop. We ...

Read More Source

April 18th, 2019

In Technology

No Comments

When you accidentally put your database on a public subnet in AWS

This feels like a very easy mistake to make:

Here’s an example where you can get into trouble. Digging through some infra code, reviewing with a new devops hire, we were going through everything with a fine toothed comb. We found that the RDS instance was being deployed in PUBLIC subnet, instead of private.

Alerted to the problem, we first checked to see whether it was accessible from the internet at large. It wasn’t, because we had not exposed a public facing ...

Read More Source

April 16th, 2019

In Technology

No Comments

Anna Kucirkova: the history of computer programming languages

This article misses all of the Lisps, and focuses mostly on Algol languages, but it does a nice overview of that branch of computer history, and how that branch gave us the syntax of modern popular languages like Javascript.

1883: That’s right, the first programming language was born in 1883. The algorithm for the Analytical Engine was created by Ada Lovelace for Charles Babbage’s Analytical Engine to compute Bernoulli numbers. This is widely considered to be the first computer programming ...

Read More Source

April 16th, 2019

In Technology

No Comments

The cult of complexity

Interesting:

Writing the same dull CRUD application over and over is boring. But what if you could figure out a way to get all the simple CRUD applications to talk to one another? That’s a nice and juicy puzzle. This perhaps explains the complexity fetish I see in so many “Enterprise” architectures and applications. Some of it is accidental complexity, accrued from years of piecing together parts that were never meant to work with one another. But I don’t think accidental ...

Read More Source

April 16th, 2019

In Technology

No Comments

Serverless architecture uses servers, requires work, and is a stupid name

For all of you who continue to close your eyes and refuse to look at reality, here is another post pointing out the work involved with “simple” setups:

Also:

A task definition defines the actual container you want to run. The problem I ran into immediately here is that this thing is insanely complicated. Lots of the options here are very straightforward, like specifying the docker image and memory limits, but I also had to define a networking model and a variety ...

Read More Source

April 3rd, 2019

In Technology

No Comments

Stream remote file via Aleph to AWS S3?

Interesting:

What I’ve got to do is to get a file stream from a remote HTTP server and then upload that stream into AWS S3.

I query a URL with http/get, parse Content-Length header to know the file size and pass that into AWS PutObjectRequest as follows:

(aws/put-object aws {:bucket-name bucket :key key ...
Read More Source

March 20th, 2019

In Technology

No Comments

When will Apple get its act together regarding iPhone development?

Anyone who has developed for both Apple and Android is aware that the process tends to be easier on Android. But why is this? Apple has a vast pool of money, so they could invest some of it to make iPhone/iPad development easier. And this is a well known path for achieving software dominance. One of the things that allowed Microsoft to win, back in the 1990s, was their focus on building tools that made it easy for developers to ...

Read More Source

March 15th, 2019

In Technology

No Comments

What is the biggest open problem in NLP?

Interesting:

Natural language understanding

I think the biggest open problems are all related to natural language understanding. […] we should develop systems that read and understand text the way a person does, by forming a representation of the world of the text, with the agents, objects, settings, and the relationships, goals, desires, and beliefs of the agents, and everything else that humans create to understand a piece of text. Until we can do that, all of our progress is in improving ...

Read More Source

March 15th, 2019

In Technology

No Comments

biLSTMs will solve all of your NLP problems

Interesting:

Using pre-trained models

We have several different English language pre-trained biLMs available for use. Each model is specified with two separate files, a JSON formatted “options” file with hyperparameters and a hdf5 formatted file with the model weights. Links to the pre-trained models are available here.

There are three ways to integrate ELMo representations into a downstream task, depending on your use case.

Compute representations on the fly from raw text using character input. This is the most general method and will handle ...

Read More Source

March 15th, 2019

In Technology

No Comments

ELMO representations are deep, contextual and character based NLP

Interesting:

ELMo is a deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy). These word vectors are learned functions of the internal states of a deep bidirectional language model (biLM), which is pre-trained on a large text corpus. They can be easily added to existing models and significantly improve the state of the art across a broad range of challenging ...

Read More Source

March 5th, 2019

In Technology

No Comments

An overview of different approaches to NLP word embeddings

It’s interesting that the simple approaches do fairly well:

Smooth Inverse Frequency

Taking the average of the word embeddings in a sentence tends to give too much weight to words that are quite irrelevant, semantically speaking. Smooth Inverse Frequency tries to solve this problem in two ways:

Weighting: like our tf-idf baseline above, SIF takes the weighted average of the word embeddings in the sentence. Every word embedding is weighted by a/(a + p(w)), where a is a parameter that is typically set ...

Read More Source

March 3rd, 2019

In Technology

No Comments

Matrix math in Clojure

This is a really great series of articles:

Post external references 1https://dragan.rocks/articles/17/Clojure-Linear-Algebra-Refresher-3-Matrix-Transformations#orgfb4c0f9 Source

February 25th, 2019

In Technology

No Comments

In Clojure, it is easy to extend the Function protocol

I guess I’ve known this for years, but this is a wonderfully clear article:

(defprotocol IFn "Protocol for adding the ability to invoke an object as a function. For example, a vector can also be used to look up a value: ([1 2 3 4] 1) => 2" (-invoke [this] [this a] [this a b] [this a b c] [this a ...
Read More Source

February 20th, 2019

In Technology

No Comments

Women as the original software developers

Interesting:

Often enough, Wilkes’s code didn’t produce the result she wanted. So she had to pore over her lines of code, trying to deduce her mistake, stepping through each line in her head and envisioning how the machine would execute it — turning her mind, as it were, into the computer. Then she would rewrite the program. The capacity of most computers at the time was quite limited; the IBM 704 could handle only about 4,000 “words” of code in its ...

Read More Source

February 20th, 2019

In Technology

No Comments

Ego driven software development is not good software development

This is the kind of emotional impulse that for thousands of years would cause men to challenge one another to a duel:

And finally, I became the exact thing I hated: a toxic asshat throwing his skills around like fists. I don’t do code review for the business, I just like showing the rookies their place. My skills have finally started to pay off.

If a guy brings me his code, and it has mistakes, it brings insane pleasure from how smart ...

Read More Source

February 2nd, 2019

In Technology

No Comments

What should you do when your machine learning algorithm is broken?

I’ve made every mistake that they mention in this lecture. Most of the lecture is devoted to diagnostics for ML, how you should act when your ML is not performing the way you expected:

Post external references 1https://see.stanford.edu/Course/CS229/36 Source

January 18th, 2019

In Technology

No Comments

Peter-Paul Koch asked for a moratorium on new Web features in 2015

I missed this at the time. Interesting:

The main counterargument is that the moratorium would stifle web innovation. Since web innovation is currently defined as “emulating native even more,” I think a bit of stifling would actually be a good idea. Once web innovation is redefined as something that’s actually about the web we can proceed as usual.

Another counterargument is “But we need feature X!” Everybody will have a favourite upcoming feature that would be hit by the moratorium — mine ...

Read More Source

December 28th, 2018

In Technology

No Comments

Please do not simplify this code

Interesting:

// ================================================================== // PLEASE DO NOT ATTEMPT TO SIMPLIFY THIS CODE. // KEEP THE SPACE SHUTTLE FLYING. // ================================================================== // // This controller is intentionally written in a very verbose style. You will // notice: // // 1. Every 'if' statement has a matching 'else' (exception: simple error // checks for a client API call) // 2. Things that may seem obvious are commented explicitly // // We call this style 'space shuttle style'. Space shuttle style is meant to // ensure that every branch ...
Read More Source

December 16th, 2018

In Technology

No Comments

But we’ve long since reached the point where coordinating our design vision by osmosis is not working well

The community around the Rust language is increasingly in conflict with itself. Why allow a team to grow out of control like this?

That’s a total of 770 comments on the design of the API for the Pin type, which has not yet been stabilized; I expect it will pass 800 comments before this is done. This is just one significant but ultimately fairly small std API addition, and it doesn’t isn’t including the discussion that’s gone on around the ...

Read More Source

December 3rd, 2018

In Technology

No Comments

Why does Clojure continue to draw so much creativity?

Clojure continues to innovate.

Navigation is generic and extensible

As explained before, the core of Specter is the defnav operator upon which Specter’s entire collection of navigators is built. Specter’s extensibility is one of its most important features, as you may use data structures other than the ones Clojure provides. As I mentioned, I do a lot of work with graphs, and I just wouldn’t be able to work with them in any reasonable way without my internal collection of graph navigators ...

Read More Source

November 19th, 2018

In Technology

1 Comment

To start with, tran­sient query surges are no longer a prob­lem?

Tim Bray has an interesting article about the future of REST. However, I fail to understand this sentence:

“To start with, tran­sient query surges are no longer a prob­lem”

Why? The external world can still send unexpected surges of traffic, yes?

Anyway, this is worth reading:

Post-REST: Mes­sag­ing and Event­ing · This ap­proach is all over, and I mean all over, the cloud in­fras­truc­ture that I work on. The idea is you get a re­quest, you val­i­date it, maybe you do some com­pu­ta­tion on ...

Read More Source

November 19th, 2018

In Technology

No Comments

Tim Bray, now at Amazon, is working on a universal language of software states

Interesting. I’ll look at this later. I wonder if this will become important, or if it will go the way of RSS/Atom, something with huge potential, that somehow never clicks?

Post external references 1Ac­cept­ed Source

November 19th, 2018

In Technology

No Comments

Circumventing the issue of mixing Futures and Actors required extra head scratching time

I’ve already linked to this article, where Movio talked about moving away from Scala and towards Go. But I just re-read and had more to say about this:

First, we made the mistake of making the `persist` function synchronized. This guaranteed that buffer-full-based invocations would not run concurrently with timeout-based invocations. However, because the stream digest and the `persist` functions did run concurrently and manipulated the buffer, we had to further synchronize those functions to each other!

In the end, we ...

Read More Source

November 19th, 2018

In Technology

No Comments

Is Scala necessary?

There is a part of me that wonders if Scala will last. For awhile it was seen as Java 2.0. It generated some good ideas. People were excited about it for awhile. But it suffered its own chaos. It was too experimental to become the new standard Enterprise language. If it fails, I wonder what will take its place?

First, we made the mistake of making the `persist` function synchronized. This guaranteed that buffer-full-based invocations would not run concurrently with ...

Read More Source

November 14th, 2018

In Technology

No Comments

Why did Basecamp crash?

Is this the best way to store events? In a relational database? Wouldn’t a log be better for this kind of data? Are the events actually relational? When not switch to a historic paradigm? Clearly, the Ruby On Rails model can be made to do almost anything, but is it ideal?

The root cause was that our database hit the ceiling of 2,147,483,647 on our very busy events table. Almost every single activity in Basecamp is tracked in this table. ...

Read More Source

November 1st, 2018

In Technology

7 Comments

Zed Shaw is angry (abstraction and indirection)

In the tech world, a headline such as “Zed Shaw is angry” is the ultimate evergreen headline, since he is always angry, and it is easy to get tired of his ranting. But I think he has a good point here:

I hate using badly designed APIs. I hate it even more when someone beats me over the head with words they were handed in some rhetoric class masquerading as a computer science course. Words like “abstract”, “pattern”, and “object ...

Read More Source

September 5th, 2018

In Technology

No Comments

The distracting temptations of beautiful strongly-typed languages

Interesting:

‘There’s something very seductive about languages like Rust or Scala or Haskell or even C++. These languages whisper in our ears “you are brilliant and here’s a blank canvas where you can design the most perfect abstraction the world has ever seen.’

Post external references 1https://news.ycombinator.com/item?id=17904580 Source

August 28th, 2018

In Technology

No Comments

Debugging a problem with an app running under Supervisord

I’ve run into this before, an app seems to work but if I run it under Supervisord it doesn’t work, so does that mean there is a problem with Supervisord or with the app itself? This is a useful bit of debugging advice:

Does the command supervisorctl status work when the process is in this state? If that hangs or doesn’t connect, it might indicate a problem in supervisord. If supervisorctl commands work normally, that almost always indicates something is ...

Read More Source

August 10th, 2018

In Technology

3 Comments

Why does anyone use Python?

Summary: Python 1.x and 2.x brought some interesting ideas to the tech world, but the upgrade to 3.x has brought an astonishing number of problems in terms of version conflicts and dependency management. Python no longer offers ease of setup, as any non-trivial system will run into version problems. If you are a CTO, you should consider using a technology stack that has a long history of being stable, such as any of the languages that run on the JVM ...

Read More Source

July 28th, 2018

In Technology

No Comments

Ghostery has killed my use of Google Chrome

For the last 2 years I had Ghostery installed on Google Chrome (on a Mac) and I was using Chrome as my main web browser. I love the way it protected me from spam advertising. I’d open my favorite magazine and a purple box would open in the bottom right part of the browser window, where Ghostery would list all the advertising widgets it had blocked. The list was eye opening for me. On some magazine sites there were as ...

Read More Source

July 23rd, 2018

In Technology

No Comments

The cost of Test Driven Development

Interesting:

“The nice thing about working at Microsoft,” Nagappan says, “is that the development organization is large enough that we could select teams that allowed for an apples-to-apples comparison. We picked three development projects under the same senior manager and looked at teams that used TDD and those that didn’t. We collected data from teams working on Visual Studio, Windows, and MSN and also got data from a team at IBM, since the project was a joint study.”

The study and its ...

Read More Source

July 19th, 2018

In Technology

No Comments

How is an operability review different from monitoring?

I read this article but I didn’t understand what was gained, compared to good monitoring pushing stuff into good graphs, thus giving people a real-time view of the system.

Architecture & Deployment diagram: The first thing is to have the deployment & architecture diagrams. Where architecture depicts the logical flow of request/ response inside the system, deployment diagram is focussed on the actual infrastructure. But if the system is not too complex, it makes sense to have them in a single ...

Read More Source

July 18th, 2018

In Technology

No Comments

Overview of the TICK stack

It’s odd that I spend all day reading tech news, and there are still so many project that I know nothing about:

This project is a very young one, only about one year old (first commit in github was on aug 2015). It is, however, backed by the world’s most popular TimeSeries database: InfluxDB. When diving in, you can see this is a very energetic group, which has covered a lot of ground in a very short time and is presenting ...

Read More Source

July 18th, 2018

In Technology

No Comments

The “index” is where Riemann keeps all of its information

Interesting:

The index is a table of the current state of all services tracked by Riemann. Each event is uniquely indexed by its host and service. The index just keeps track of the most recent event for a given (host, service) pair.

Streams run quickly and retain as little state as possible. The index is Riemann’s state table; its picture of the world. The dashboard, network clients, and streams can all query the index to see what the system looks like now. ...

Read More Source

July 18th, 2018

In Technology

No Comments

What does ‘cider-jack-in’ do

I was a bit curious about this::

When you ask Emacs to start a new REPL using cider-jack-in it actually sends this request to Leiningen, the Clojure build automation tool. Once Leiningen has started the REPL, then a connection is made to that REPL from Emacs using the nREPL protocol.

This feature is from the CIDER package for Emacs, a Leiningen plugin called cider-nrepl and the Clojure tools.nrepl library are required.

Cider configuration (version 0.10 and earlier) In earlier versions of CIDER ( Source

July 18th, 2018

In Technology

No Comments

Details on AWS private/public subnets

A note to myself. I’m doing this with Terraform, but of course, one has to know how it will all actually work. Connecting a private subnet to the public world:

Resolution

1. List the Availability Zones that have the instances you want to attach to the load balancer.

2. Create an equal number of public subnets in the same Availability Zones where your private instances exist. To ensure that the load balancer can scale properly, verify that each subnet for the load balancer ...

Read More Source

July 15th, 2018

In Technology

No Comments

Use Packer to build AMIs that use ZFS for the file system

This is a great article: Building ZFS Root Ubuntu AMIs with Packer. You can do the same thing with Docker but I would not want my software engineers taking on those devops tasks. Indeed, one reason I’m so hostile to Docker is because it disrupts the specialization of tasks that I want to see on my teams. I expect to work with great devops people, and I’ll leave devops decisions to them.

From the article:

For all applications of importance or ...

Read More Source

July 15th, 2018

In Technology

No Comments

Things that allow you to recover from having a bad tech team

Over the years, I’ve come to realize that there are 2 failsafes that make it easier for an early stage technology startup to recover from making bad hires on the tech team.

1.) Hire a great project manager. A great project manager is able to get good work out of mediocre programmers. Also, great project managers have a good intuitive feel for how long tasks should take, so if some task misses its deadline, the project manager will know why. ...

Read More Source

July 14th, 2018

In Technology

No Comments

Elasticsearch Backup : Snapshot and Restore on AWS S3

Because Terraform will delete all of your databases when you rebuild your infrastructure, and that is a nasty shock. So you should manually back up ElasticSearch:

In Aws you have many options to allow [users|servers] access AWS resources. The idea we have is to setup a snapshot repository on Amazon Aws S3 and doing a restore from that specific location. In order to do that, the servers that have to access S3 must be authorised. Thankfully to Amazon we have many ...

Read More Source

July 12th, 2018

In Technology

No Comments

Nasty moments in Open Source development

Two bad incidents, in PHP and Python.

Escalate? Oh how I wish I had someone to escalate to. Painful to read. Here is someone filing a bug report and attacking Ramus Lerdorf, the guy who created PHP.

[2010-01-08 21:51 UTC] endosquid at endosquid dot com This is going to cause us MONTHS or fixing code for no real benefit since this behavior change is arbitrary and seemingly, was made for no reason. We are all engineers and developershere, and ...

Read More Source

July 10th, 2018

In Technology

No Comments

A meta technology hype roadmap

Very good:

Let me write a meta technology hype roadmap, so we can place these sorts of articles: * Old technology is deemed by people too troublesome or restrictive.

* They come up with a new technology that has great long-term disadvantages, but is either easy to get started with short-term, or plays to people’s ego about long-term prospects.

* Everyone adopts this new technology and raves about how great it is now that they have just adopted it.

* Some people warn that the ...

Read More Source

June 20th, 2018

In Technology

No Comments

Mobile operators appear to have increasingly turned to new techniques for tracking users — HTTP header enrichment

Interesting:

Our analysis reveals a range of HTTP headers injected into mobile user traffic by 13% of the 299 mobile operators in our dataset (including both MNOs and MVNOs). We classify the headers in our dataset into three categories, based on their likely purpose:

• Privacy-compromising headers (5 operators): HTTP headers leaking sensitive information that can uniquely identify the device (IMEI), the subscriber (MSISDN or phone number), or the subscriber’s location.

• Tracking headers (6 operators): operator-generated UIDs (subscriber-unique identifiers) that enable user ...

Read More Source

June 13th, 2018

In Technology

No Comments

Deep Learning Needs Clojure – Carin Meier

Source

June 13th, 2018

In Technology

No Comments

Jenkins pipelines must be built so they can run across multiple executors, even if you only run Jenkins with one executor

A bit hidden, and important for anyone using Jenkins, This fact should be better known:

Serialization and the “node” Block Jenkins pipelines must be built so they can run across multiple executors, even if you only run Jenkins with one executor. This is done using the node block. With it, a job can be split into smaller pieces that can be scheduled to run on any executor on any agent in the cluster. Once a node block completes, Jenkins serializes the ...

Read More Source

May 30th, 2018

In Technology

No Comments

SELinux provides tools to more finely control the activities allowed to users, processes, and daemons

Interesting:

SELinux provides tools to more finely control the activities allowed to users, processes, and daemons to limit the potential damage from vulnerabilities.

In the third and final part of our server security series, we will look at how we can enhance the security of Linux-based AWS EC2 instances with SELinux. We will learn how to set up SELinux on Amazon Linux, and we will walk through a simple example on Red Hat Enterprise Linux (RHEL).

In Linux, we can easily control access ...

Read More Source

May 17th, 2018

In Technology

No Comments

Markov Decision Process versus Reinforcement Learning

I’m still struggling to learn when to use the various different Machine Learning approaches:

To solve a problem like this, you can use MODEL-BASED approaches if you know how likely it is that the robot will move from one state to another (that is, the transition probabilities for each action) or MODEL-FREE approaches (you don’t know how likely it is that the robot will move from state to state, but you can figure out a reward structure).

Markov Decision Process (MDP) – ...

Read More Source

May 17th, 2018

In Technology

No Comments

Sean Hull offers a great tutorial regarding Terraform and Amazon’s Elastic Container Service

This is very much worth reading if you are using or considering Terraform:

It takes a bit of getting used to, but This terraform how to, should get you moving. You need an EC2 host to run your containers on, you need a task that defines your container image & resources, and lastly a service which tells ECS which cluster to run on and registers with ALB if you have one.

If you don’t know, the ALB is the Application Load Balancer. ...

Read More Source

May 17th, 2018

In Technology

No Comments

Impressive things about Terraform

I am impressed with the way that Terraform allows us to specify a setup of servers and load balancers and firewalls and whitelists and set it up, and use it for development, and then the day we decide to go public, we only have to run one command to re-create everything in production. That is a high level of automation.

Good people have insisted that this is only possible while using Docker containers, but I don’t see that at all. ...

Read More Source

May 16th, 2018

In Technology

No Comments

mountebank is the first open source tool to provide cross-platform, multi-protocol test doubles over the wire

This is interesting. I mostly just use the “curl” command to test APIs over the Internet, but this could be useful when I need a lot of mock services:

mountebank employs a legion of imposters to act as on-demand test doubles. Your test communicates to mountebank over http using the api to set up stubs, record and replay proxies, and verify mock expectations. In the typical use case, each test will start an imposter during test setup and stop an imposter ...

Read More Source

May 13th, 2018

In Technology

45 Comments

Docker is the dangerous gamble which we will regret

Also read: “My final post regarding the flaws of Docker and Kubernetes and their eco-system“

Summary: don’t use Docker, or any other container technology. Use Terraform and Packer instead. It’s one less level of virtualization, and therefore one less level of complexity.

There is perhaps one good argument for using Docker. It is hidden by the many bad arguments for using Docker. I’m going to try to explain why so much Docker rhetoric is stupid, and then look at what reason might ...

Read More Source

May 13th, 2018

In Technology

12 Comments

Docker protects a programming paradigm that we should get rid of

Do you know about giraffes? In particular, their recurrent laryngeal nerve? Here is the deal: the nerve used to go straight from the brain to larynx, and in fish this is a short, direct connection, with the nerve passing behind the gills, but as fish evolved into creatures that lived on land, the gills got pulled into the body of the creature, and became what we call lungs. As that evolution happened, the nerve got pulled further and further into ...

Read More Source

May 13th, 2018

In Technology

3 Comments

The Coral Reef Pattern of incremental improvement

The Acme Legal Firm needed a small office, and there was little available for rent in their city, so they decided they would build their own place. They hired a builder and asked her what they should do.

“Well, I think we should build this office with wood. I’m a big believer in wood. It is beautiful and fairly strong, and it is easy to hire workers who know how to work with wood. The failure modes of wood are ...

Read More Source

May 12th, 2018

In Technology

No Comments

How much pain does it take before Kubernetes is not worth it?

Interesting:

For our particular project, I think we’re still getting started when you take into consideration the likely overall lifetime. Still, I suspect that moving an existing architecture wholesale to Kubernetes is a bad idea in general; if you’ve not grown up within its particular limitations, they will prove to be an endless annoyance4, even if there are a host of other tangible benefits over going solo. This may not be limited to Kubernetes, it may just be true for anything, ...

Read More Source

May 12th, 2018

In Technology

No Comments

How to change complex network systems once you’ve made important decisions?

Interesting:

Lets use service discovery as an example. We’ve used service discovery along with health checks and self diagnostics to be able to do some fairly interesting things during outages and, more importantly, slowdowns. These are hard problems, and we’ve solved them rather crudely in places, but these solutions still give us a level of proven, necessary sophistication for system stability.

Unfortunately, our approach is slightly incompatible with Kubernetes’ centralized one. Usually, when this happens, your needs are just too sophisticated, ...

Read More Source

May 12th, 2018

In Technology

No Comments

Why programmers are biased to the stuff they already know

Interesting:

In the real world, these biases are amplified because “things we build ourselves” is actually “things we already know”, and this includes “things we already learned”. I suspect that a significant source of programmers flooding into management is the decreasing wavelength of full scale re-education on how to accomplish the exact same thing slightly differently1 with diminshing returns on tangible benefits.

I’ve been “learining” Kubernetes for a significant migration we are attempting at $WORK. I’ve found it tough going. It’s a ...

Read More Source

May 11th, 2018

In Technology

No Comments

Systems of process orchestration

I get that Linux Cgroups might be acceptable, in the sense that “perfect is the enemy of the good” and Cgroups are maybe good enough for now. I’d prefer it if the industry could rethink what a multi-user computer is, get rid of Unix, take the best ideas forward into a new system, and build something where something as clunky as Cgroups and Docker are not needed — it should be possible to have a “fat process” which does what ...

Read More Source

May 11th, 2018

In Technology

No Comments

What are Linux cgroups

This is not the way I would have gone, but whatever. Here we are now:

cgroups (abbreviated from control groups) is a Linux kernel feature that limits, accounts for, and isolates the resource usage (CPU, memory, disk I/O, network, etc.) of a collection of processes.

Engineers at Google (primarily Paul Menage and Rohit Seth) started the work on this feature in 2006 under the name “process containers”.[1] In late 2007, the nomenclature changed to “control groups” to avoid confusion caused by multiple ...

Read More Source

May 8th, 2018

In Technology

No Comments

So many things can go wrong with Docker

Sean Hull always has interesting things to say, and so it is worth reading what tripped him up with a recent project where he had to Dockerize everything:

When a service is run, ECS wants to have *all* of the containers running together. Just like when you use docker-compose. If one container fails, ecs-agent may decide to kill the entire service, and restart. So you may see weird things happening in “docker logs” for one container, simply because another failed. What ...

Read More Source

May 7th, 2018

In Technology

No Comments

Is there anyone who can keep up with everything AWS offers?

As I’ve suggested before, unless you a full time specialist tracking AWS, you won’t be able to keep track of all the services coming out of AWS. The one I just learned about is ALB:

Amazon Web Services (AWS) just announced a new Application Load Balancer (ALB) service.

I spent some time playing with the new service to understand what it offers and to see how it fits into our cloud architecture.

In summary, ALB is a massive improvement over ELB in almost ...

Read More Source

April 30th, 2018

In Technology

No Comments

Confluence is very difficult to set up

I had endless errors with Confluence. Each time I started over, I set up a new EC2 instance. I eventually learned that it was possible to restart the setup process by deleting the file confluence.cfg.xml so I did this:

/etc/init.d/confluence stop

rm /var/atlassian/application-data/confluence/confluence.cfg.xml

/etc/init.d/confluence start

but now I’m trying to finish the setup, and all I get is a blank page:

Does anyone know how to fix this?

Source

April 12th, 2018

In Technology

No Comments

Excessive abstraction in PHP frameworks

I’m working with Laravel now, a PHP framework. I see this on StackOverflow:

With Laravel, everything is more complex than it needs to be. Yesterday I was dealing with the issue of logging and errors. How to get Laravel to output errors somewhere that I could see them? Well, that involves overriding some logging classes. The tone of the documentation is always “See, it’s very simple, and flexible, just override the default classes, inherit from the parent, override the method, choose ...

Read More Source

April 9th, 2018

In Technology

No Comments

Do the staff at Jezebel manually manage the trolls in the comments?

Apparently so, this is interesting.

Someone named Jinni writes:

Our Muqaddimah of Code 45* is still banned, alas. A bright point in my own days feels notably dimmed. For those of you who didn’t read (or quite ‘get’) Code 45*; it was a hallucinatory tour into the subconscious of Donald Trump as evidenced by the odd and illogical punctuation of his disordered tweets. Code 45*, like a creative, and slightly mad psychotherapist, attempted to wrest vestiges of meaning from the morass of ...

Read More Source

April 8th, 2018

In Technology

No Comments

PHP code for Apple Push Notification Service

I just lost a week of my life trying to send a message from my server to the APNS servers. A previous team had given us some bad PHP code which I could not get working. I finally tore it out and replaced it with the example I found on StackOverflow:

/* We are using the sandbox version of the APNS for development. For production environments, ...
Read More Source

April 8th, 2018

In Technology

No Comments

A pem tool to generate the pem file for Apple Push Notification Service

Having lost the last week of my life trying get a pem file that will let me talk to APNS, I will link to this, because it seems to be a solution for the future:

Automatically generate and renew your push notification profiles

Tired of manually creating and maintaining your push notification profiles for your iOS apps? Tired of generating a pem file for your server?

pem does all that for you, just by simply running pem.

Post external references 1https://docs.fastlane.tools/actions/pem/ Source

April 6th, 2018

In Technology

No Comments

Why don’t any articles about Apple Push Notifications agree about what commands to use?

Apparently there are a lot of ways to do this? I am suspicious, because I lost days trying to get this to work. This article says:

Now we need to encode these two files in Base64:

openssl base64 -in cert.pem -out cert.txt

openssl base64 -in key.pem -out key.txt

But this gist says:

Step 1: Create Certificate .pem from Certificate .p12

Command: openssl pkcs12 -clcerts -nokeys -out apns-dev-cert.pem -in apns-dev-cert.p12

Step 2: Create Key .pem from Key .p12

Command : openssl pkcs12 -nocerts -out apns-dev-key.pem -in apns-dev-key.p12

None ...

Read More Source

April 5th, 2018

In Technology

No Comments

Apple Push Notifications converting p12 to pem files

None of the commands listed on this page worked for me, and yet that is the topped ranked page in Google. What did work for me:

openssl pkcs12 -in apns_certificates.p12 -out apns_certificates.pem -nodes openssl pkcs12 -in apns_key.p12 -out apns_key.pem -nodes chmod 0400 apns_key.pem chmod 0400 apns_certificates.pem openssl s_client -connect gateway.sandbox.push.apple.com:2195 -cert apns_certificates.pem -key apns_key.pem cat apns_certificates.pem apns_key.pem > Peeroapp.pem Post external references 1https://gist.github.com/shahdhiren/9ca059ac0762f7ef0fcf71a79ed5b022#file-p12topem-txt-L5 Source

April 3rd, 2018

In Technology

No Comments

The difficulties of starting a career in tech

This is a good summary of some issues with starting a career in tech. This one jumps out at me, though I realize the issue of portfolios is controversial in some quarters.

More than a couple of times have I met developers looking for web development gigs without a proper portfolio to show off.

Having a portfolio helps to demonstrate what you are capable of doing.

You’ve probably been told that having a personal domain puts you a mile ahead in your ...

Read More Source

March 27th, 2018

In Technology

No Comments

I just lost a week of my life dealing with one bug in a Cordova/Ionic mobile app

Over the last week I had two marathon days where I worked more than 12 hours, and I had 4 normal days where I worked 6 to 8 hours, and all of it was wasted on one bug.

I’ve a friend who wanted an app built that would work for both iOS and Android. He contracted with a team in India to build it. They used Cordova and Ionic.

When they were done with the project they sent it to ...

Read More Source

March 25th, 2018

In Technology

No Comments

Apple takes two weeks to update a forgotten password?

I am amazed by this. On Friday I tried to log into:

https://developer.apple.com

None of my regular passwords worked. Only later did I realize I’d probably set up a separate account for this, using some unusual password. Anyway, I figured this would not be a big deal. I clicked “I forgot password”. I put in my email and then my phone number.

Apple then sent a text message to my phone, with a code that I was to type into the website. ...

Read More Source

March 22nd, 2018

In Technology

No Comments

Even a basic iOS app generates a huge number of errors in Xcode

I’ve decided I should learn how to build native iOS apps. So I am following a basic tutorial for Xcode and iOS.

As soon as I’ve created the project, I try to run it. It does compile and run, but the log is full of hundreds of errors. I’m putting a small sampling below.

Does Apple realize how confusing this is for someone who is just beginning?

I learned how to program back in 1999, writing PHP scripts. One great ...

Read More Source

March 19th, 2018

In Technology

No Comments

CSS is terrible because it is global state

CSS is broken because the style sheets are global. Once upon a time, this didn’t seem terrible because it was thought that CSS would operate as CONSTANTS. But they are not constants. On any modern website or app, the styles are always changing. So the styles are really a global variable, and they have all the problems that 60 years of computer science has taught us to expect from global variables.

There are other problems with CSS. Perhaps the biggest ...

Read More Source

March 19th, 2018

In Technology

No Comments

Online education in the 1960s

Interesting:

As Dear tells it, students were far more interested in using PLATO to chat with one another than to complete their lessons. They even created their own emoticons by taking advantage of a feature within PLATO’s operating system that allowed them to stack letters on top of each other instead of typing them side by side. Typing the word “WOBTAX” in a stack produced a smiley face.

“This was a great pastime of many undergraduates, many of whom would flunk out ...

Read More Source

March 10th, 2018

In Technology

No Comments

The lack of volume in a high-dimensional cube

I wish this felt more intuitive to me:

This is a starting point for intuition:

To provide some intuition consider the situation in two dimensions, as shown in Figure 10. For a point on the circle to be close to the equator, its y-coordinate must be small.

So, this is sort of an expression of the Curse Of Dimensionality?

But perhaps the most amazing thing is that we can not figure out how many spheres might surround a central sphere, in such a ...

Read More Source

March 8th, 2018

In Technology

No Comments

In Apache you can setup a reverse proxy with either RewriteRule or ProxyPass

With Apache I have always done something like this:

ProxyPass / http://127.0.0.1:8080/ ProxyPassReverse / http://127.0.0.1:8080/

But I just saw this interesting bit on StackOverflow:

RewriteEngine On RewriteRule ^(/.*)$ http://app-server:8181/jellyfish$1 [P] ProxyPassReverse / http://app-server:8181/jellyfish/

Which does the same thing. The RewriteRule says “grab everything after the ...

Read More Source

February 12th, 2018

In Technology

No Comments

Containers like Docker will not fix the sick politics of your company

Interesting:

Deciding what’s yours and what’s theirs is the basis of every sibling-rivalry détente. In Eric Brewer’s CAP theorem you can pick two of consistency, availability, and partition tolerance as long as one of them is partition tolerance, because, as distributed systems expert Caitie McCaffrey puts it, “physics and math.” In a distributed system that contains humans in multiple time zones, you’re inevitably going to have partitions, and waiting 10 hours for headquarters to wake up and make a decision is ...

Read More Source

February 3rd, 2018

In Technology

No Comments

When things go wrong on a software project

I like this:

Whenever things are not progressing as expected, it is a natural instinct to fall back to micromanagement, but it seldom truly cures the ill. It’s probable that your team disagrees with your vision or its feasibility – and that you’re either not listening to their feedback, or they don’t think you’d care. It’s good to assume that most of your employees are as smart or smarter than you; barking your orders at them more loudly or more frequently ...

Read More Source

February 2nd, 2018

In Technology

No Comments

A very long debate about when to use a library and when to write your own code

The whole thing is interesting, but Jens Stimpfle’s comment comes close to representing my view:

“I see the value in learning by building yourself, but from a software engineering point of view, using a tried and tested framework is likely to give you higher quality product in less time.”

Well, the framework I was to use last (Qt) had bugs that simply can’t be fixed by users (memory leaks, double free leading to segfault when exiting after reloading QML engine) and immature ...

Read More Source

January 18th, 2018

In Technology

No Comments

The continued adaptation, modification and correction of errors in them, is essentially dependent on a certain kind of knowledge possessed by a group of programmers

Interesting:

The conclusion seems inescapable that with at least certain kinds of large programs, the continued adaptation, modification and correction of errors in them, is essentially dependent on a certain kind of knowledge possessed by a group of programmers who are closely and continuously connected with them.

Post external references 1http://pages.cs.wisc.edu/~remzi/Naur.pdf Source

January 17th, 2018

In Technology

No Comments

Chris Zheng is angry at the Clojure community

This initially sounds like it is going to be interesting:

I say it with much admiration and respect to all the members of community. I’ve learnt so much from being in this microcosm of dynamism, ideas and learning over the past 8 years. Clojure has allowed me to get to know so many amazing people, to travel to a whole bunch of places and to do things that I had never thought possible.

Having said that, I’ve gone from a wide-eyed, ...

Read More Source

January 16th, 2018

In Technology

No Comments

What is up with Narrative.io?

When I go to the website of Narrative.io all I see is black. Is this because I have an ad blocker installed? If so, this is the most dramatic ad-block caused bug that I’ve seen. Something like 20% of everyone on the Web is now using ad-blocking. Narrative.io should keep that in mind when they design their site.

Post external references 1http://www.narrative.io/ Source

January 6th, 2018

In Technology

No Comments

Yegor Bugayenko: 5 things to improve Java

Yegor Bugayenko offers suggestions for making Java better:

Global Variables. There are Singletons in Java, which, as we all know, are nothing but global variables. Wouldn’t it be great to enable global variables in Java and get rid of Singletons. PHP, JavaScript, Ruby and many other languages have them, why doesn’t Java?

Global Functions and Namespaces To group static methods together we create utility classes, where we have to define private constructors to prevent their instantiation. Also, we have to remember which ...

Read More Source

January 5th, 2018

In Technology

No Comments

How should your company handle data safety and version control?

I’ve been writing software professionally since 1999. The first few years, I didn’t work at a company that used version control. Colin Steele mentioned Subversion to me in 2005. I used it till 2011. Since 2011, every company I’ve worked at has used Git.

From 2005 to 2011 we used Subversion to keep track of all projects. This was at both startups that I helped run, and also while doing most client work. During these years, I only had one ...

Read More Source

January 5th, 2018

In Technology

No Comments

The problem with HTML

(originally from a longer essay)

Just to offer some perspective about where I’m coming from, I’ll start by saying that I think of these as tools that made it easy for beginners to be productive:

Hypercard

Visual Basic (obviously I mean the classic versions, before .NET. The stuff in the 1990s was genius)

Adobe Flash

Adobe Dreamweaver

(Of these links, I especially recommend the one about Hypercard.)

I spent 1995 learning HTML and putting together simple websites. I was thinking this is something I’d like to ...

Read More Source

December 26th, 2017

In Technology

No Comments

Jakub Holý on the awesome productivity of Clojurescript

I have found the REPL useful when working with Clojure. I plan to write about this soon. Jakub Holý has a great write-up regarding both Clojure and Clojurescript:

Aside – about us: Telia Norge is Norway’s second largest mobile operator. Our team has spent the past three years with Node.js, React, and (eventually) Redux creating a great webshop (for hardware and subscriptions) to replace an off-the-shelf one, with a great success. Now we set out to make also the lives ...

Read More Source

December 25th, 2017

In Technology

No Comments

Jakub Holý: how to build a shopping cart

Interesting:

However there are other concerns than simplicity and time-to-production that we could have been taken into account, I believe. But I failed to communicate those in time. The current solution would have been perfect if the network transfer times were negligible and if the backend service was 100% reliable. None of that is true. Every call takes at least 300ms and the backend can be overloaded or (as yesterday) unavailable due to network issues. I there is a high ...

Read More Source

December 25th, 2017

In Technology

No Comments

Jakub Holý: don’t add unnecessary checks to your code

This is a very good point:

Imagine you need to call .indexOf on product_code. You might be tempted to add

if (!_.isString(product_code)) return false;

Now you are safe and your code can’t fail. But when I read this code, I immediately think:

Oh, I believed product_code to be always a string but obviously it is not always the case. It must have happened in the past or the author wouldn’t have added this check. I wonder what are the possible non-string values. And what ...

Read More Source

December 18th, 2017

In Technology

No Comments

The penalty of saying something controversial on HackerNews

I made a comment on HackerNews and it was heavily downvoted. I made a good point about the limitations of Git, but people did not want to hear it. This raises an interesting issue about voting systems, in general, is there any way to find stuff that is true, reliably, when the voters might have incorrect beliefs? I think Slashdot went furthest, they took voting systems seriously, and they tried to weight the votes of those who were themselves upvoted. ...

Read More Source

December 12th, 2017

In Technology

No Comments

The Krubner Law Of Ugly Code

The Krubner Law Of Ugly Code states that any computer programming language that has a “return” keyword will tend to be ugly. “Tend” means the tendency is there, though of course, as I’m sure hundreds will rush to remind me, you can write beautiful code in any language.

Here is a simple example from a well-respected project. Here the return is used in a guard clause. This is good code, for the Java language, in that it puts the guard ...

Read More Source

December 12th, 2017

In Technology

No Comments

To what extent is “DLL Hell” a failure of interface contracts to work adequately in practice?

(Tonight I’m dumping some of my notes onto my blog. Not sure why I kept these for years without publishing. I was probably waiting for a moment when I could write a longer essay about them.)

http://www.artima.com/intv/interop.html

Bill Venners: To what extent is “DLL Hell” a failure of interface contracts to work adequately in practice? If everyone fully understands and adheres to the contract of the functions of a particular DLL, shouldn’t updating that DLL in theory not break any code?

Anders Hejlsberg: ...

Read More Source

December 12th, 2017

In Technology

No Comments

Camille Fournier (Rent the Runway) – Consensus Systems for the Skeptical Architect

(Tonight I’m dumping some of my notes onto my blog. Not sure why I kept these for years without publishing. I was probably waiting for a moment when I could write a longer essay about them.)

ZooKeeper owns your availability. Worth watching. I thought it was amazing when she said that even 5 years ago, even at Google, most programmers were not thinking in terms of distributed computing. Instead, its been the boom in microservices that has driven the adoption of ...

Read More Source

December 12th, 2017

In Technology

No Comments

Sam Newman’s book Building Microservices

I’m posting here some quotes I like from this book:

Time for a cautionary tale. Back in 2006, I was working on building a pricing system for a bank. We would look at market events, and work out which items in a portfolio needed to be repriced. Once we determined the list of things to work through, we put these all onto a message queue. We were making use of a grid to create a pool of pricing workers, allowing ...

Read More Source

December 12th, 2017

In Technology

7 Comments

Why I prefer dynamic-typing over static-typing: the speed of adapting to change

I am a fan of dynamic typing. So is corporate America. The widespread use of PHP, Ruby, Python and Javascript suggest that dynamic typing is useful. Static typing tends to be used in areas where legal regulations create needs that outweigh programmer productivity. If static typing lead to greater programmer productivity (via a reduction in bugs) then corporate America would only use statically-typed languages. But it doesn’t.

In How ignorant am I, and how do I formally specify that in ...

Read More Source

December 11th, 2017

In Technology

No Comments

Watching the AI bots on a game fight each other is like testing a data analysis model

I was researching something else and I followed a link to the game Wesnoth. I downloaded it and spent some time testing it. The game is entirely built by volunteers. The game is slow and somewhat tedious, but they did a great job with the art and they did a great job enabling a variety of scenarios.

They claim they put some time into the AI engine that lets bots play a character. I was curious about that. I set ...

Read More Source

November 25th, 2017

In Technology

No Comments

That’s supposed to be an umlaut in the title

This sounds like a very serious criticism of WordPress:

Yes, I know that’s supposed to be an umlaut in the title. I just can’t persuade WordPress to do it.

As far as I know, WordPress has handled UTF8 for a long time. Does anyone know what is going on?

Post external references 1https://krugman.blogs.nytimes.com/2017/11/24/schroedingers-tax-hike/ Source

November 25th, 2017

In Technology

No Comments

VisualBasic could have saved Microsoft Excel

Companies are rebelling against Excel:

Finance chiefs say the ubiquitous spreadsheet software that revolutionized accounting in the 1980s hasn’t kept up with the demands of contemporary corporate finance units. Errors can bloom because data in Excel is separated from other systems and isn’t automatically updated.

Older versions of Excel don’t allow multiple users to work together in one document, hampering collaboration. There is also a limit to how much data can be pulled into a single document, which can slow down analysis.

“Excel ...

Read More Source

November 2nd, 2017

In Technology

No Comments

Denormalization to diverse apps is the correct path forward for media companies

I am very pleased to see that the New York Times now has this cutting edge architecture, which almost certainly the right way forward at this time:

The denormalized log and Kafka’s Streams API

The Monolog is great for consumers that want a normalized view of the data. For some consumers that is not the case. For instance, in order to index data in Elasticsearch you need a denormalized view of the data, since Elasticsearch does not support many-to-many relationships between ...

Read More Source

October 31st, 2017

In Technology

No Comments

Matthew Vagnoni suggests the blockchain can resolve the issue of patient consent

Back in 2015, when I worked at Rollio, we hired Matthew Vagnoni for 2 weeks to come and teach us about Natural Language Processing. Vagnoni is a fantastic person to work with: he is a true expert in his field, yet he manifests very little ego about his extraordinary skill. He was friendly and calm and also authoritative, as he walked us through the various strategies we might take, when connecting to Salesforce, using NLP.

I only recently realized that ...

Read More Source

October 31st, 2017

In Technology

No Comments

XLSX2CSV in Java or Clojure

This page does not seem to rank very high on Google, which makes it difficult to find a good example of how to use Java to convert an Excel file to CSV. Last year, when I worked on this, I recall it took me many days of searching before I found this example.

Also good:

http://svn.apache.org/repos/asf/poi/trunk/src/examples/src/org/apache/poi/ss/examples/ToCSV.java

Post external references 1https://svn.apache.org/repos/asf/poi/trunk/src/examples/src/org/apache/poi/xssf/eventusermodel/XLSX2CSV.java Source

October 27th, 2017

In Technology

1 Comment

A developer asks for help, then gets angry at those who help them

This is pretty wild. Some idiot named “Amo” asks why Supervisord is using 100% of the CPU. A fellow named Paul Calabro then gives what sounds like the correct answer. Instead of saying “Thank you” Amo then threatens to downvote Calabro. Either Amo is vicious or they completely misunderstood the answer. Either way, Amo is an idiot.

If I was Amo, I would delete the question, and thus make it private, because the whole thing is embarrassing. In case ...

Read More Source

October 25th, 2017

In Technology

4 Comments

Why would anyone choose Docker over fat binaries?

This essay is obsolete. Instead, read: “My final post regarding the flaws of Docker and Kubernetes and their eco-system“

[ [ In this essay, I use the phrase “fat binary” to refer to a binary that has included all of its dependencies. I am not using it to refer to the whole 32 bit versus 64 bit transition. If I was only writing about the world of Java and the JVM, I would have used the word “uberjar” but ...

Read More Source

October 24th, 2017

In Technology

No Comments

You can set environment variables from the Supervisord config

I often tell clients to consolidate their config files, from all their various apps, inside of a folder inside of /etc (I assume Linux servers).

But an interesting alternative is to set the config in the Supervisord config, and thus consolidate all logging and config info in one place.

[program:django] environment = SITE=domain1, DJANGO_SETTINGS_MODULE=foo.settings.local, DB_USER=foo, DB_PASS=bar command = python manage.py command

One could simply have different entries for development, certain ...

Read More Source

October 24th, 2017

In Technology

No Comments

Infinite scroll is a terrible idea 99% of the time

Pinterest introduced the modern infinite scroll, so women would keep scrolling their home page. For a while, Pinterest was the fastest growing large site in the world. Infinite scroll is useful if you’re simply encouraging people to browse over random items. It is a terrible design idea if you are trying to communicate information. As Edward Tufte says, long lists are fantastic if you are trying to communicate choices or many types of information.

So check out the Web ...

Read More Source

October 15th, 2017

In Technology

No Comments

The resulting passive-aggression kills productivity faster than any methodology or technology decision

These are the truest 6 paragraphs that I’ve ever read about software development ideologies:

Whether a methodology works or not depends on the criteria: team productivity, happiness, retention, conformity, predictability, accountability, communication, lines per day, man-months, code quality, artifacts produced, etc. Every methodology works if you measure the right thing. But in terms of the only measurement that really matters—satisfying requirements on time and within budget—I haven’t seen any methodology deliver consistent results.

In his 2003 thesis People and methodologies in ...

Read More Source

October 15th, 2017

In Technology

No Comments

What are the benefits of the fictions that guide software development?

Very interesting and well written:

I’m not going to beat up on any of these paradigms, because what’s the point? If software methodologies didn’t exist we’d have to invent them, because how else would we work together effectively? You need these fictions in order to function at scale. It’s no coincidence that the Agile paradigm has such a quasi-religious hold over a workforce that is immensely fluid and mobile. (If you want to know what I really think about software ...

Read More Source

October 11th, 2017

In Technology

No Comments

NLP challenges: word ordering reverses meaning

Another good example:

When words are used in conjunction with each other, they express an idea that is bigger or more meaningful than each word in isolation. The two clauses I’m not happy I’m working and I’m happy I’m not working contain the sames words, in close proximity, but have quite different meanings.

Post external references 1https://www.elastic.co/guide/en/elasticsearch/guide/current/shingles.html Source

October 5th, 2017

In Technology

1 Comment

AWS does not protect you from devops

Once upon a time, people made fun of Linux GUIs because they would give you a graphical way to edit a configuration file, but you still had to know everything in the configuration file, and what the settings did. People made the reasonable point that slapping a pretty graphical interface on a configuration file did not make it any easier to work with Linux.

Nowadays, the clients I interact with say things like “We want to use AWS because then ...

Read More Source

September 20th, 2017

In Technology

No Comments

Where should a spec be defined relative to a function

This is an interesting conversation on the Clojure mailist, and this question is relevant for more than just Clojure. It touches on the main question of language design.

Didier feels it should be possible to define the spec with the function, and this functionality should be added to the Clojure core, so that this would become idiomatic Clojure:

Gary Trakhman argues that this would make the core “defn” macro too slow, and therefore everything about spec should be left to ...

Read More Source

September 18th, 2017

In Technology

No Comments

How to use Zach Tellman’s Clojure library Manifold

These examples are great, and I wish this was more highly ranked in Google, so I’m linking to it.

Post external references 1https://gist.github.com/aengelberg/0336196eff53550c5fb17a8ba5b535f7 Source

September 11th, 2017

In Technology

No Comments

Niche facts that Google can’t find

Google works so well that I’m shocked when it doesn’t work. I recall reading an early RFC, from the early 1970s, when someone proposed allowing variable length bytes. I tried searching Google for this. No luck. Google only shows me the famous RFCs, not the failed experiments.

Post external references 1https://www.google.com/search?q=rfc+variable+packet+length+request+for+comment&oq=rfc+variable+packet+length+request+for+comment&gs_l=psy-ab.3...11940.13794.0.14080.9.9.0.0.0.0.129.733.0j6.6.0....0...1.1.64.psy-ab..3.0.0._hLLsQhnqm8 Source

September 11th, 2017

In Technology

No Comments

Why can’t modern tools match Smalltalk?

I often have the impression that Smalltalk in the late 80s and early 90s hit a peak that has yet to be matched. Great effort was instead put toward languages that are less intelligent, and which limit the computer programmer. Does this have something to do with the deskilling of computer programming, as Stanislav says?

“It amazes me just how blindly complacent programmers have been in the face of the ongoing and very successful deskilling of their profession. ...

Read More Source

September 8th, 2017

In Technology

No Comments

The limitations of decision trees

This article has some nice setups with the visuals:

If you are going to buy a house, you will make a list of pros/cons. That much is common sense. It is also an implicit decision tree. Most of these concepts are common sense. But I am struck, yet again, and how quickly this can seem to be unimaginably complex, once you involve the abstract jargon of math:

And once again, I note the sheer awfulness of math notation. How much faster ...

Read More Source

September 8th, 2017

In Technology

No Comments

Gini Impurity is quick to measure and easy to adapt to

I’m trying to convince a client that we need to match records using a Random Forest approach. They have hit the limits of what can be done with simple string matching. I like basic articles like this, for explaining things clearly, and making it obvious how easy Random Forests can be (especially compared to Neural Nets).

Which is the better split? This is a subjective question. In practice, people use different metrics for evaluating splits. The most commonly used metric ...

Read More Source

September 8th, 2017

In Technology

No Comments

Logs are the future of software

Not sure why it has taken so many decades to become obvious, but clearly a history of all events is better than a relational database for keeping track of data. There is no reason why any company should ever use a relational SQL database. The source of truth should be the log, and one’s consumers should build their own denormalized datasets — and these will often work best in a document store, rather than a relational database.

I am very ...

Read More Source

September 7th, 2017

In Technology

No Comments

Peter Williams on Humane Interfaces

Still caught by the nostalgia I mentioned in the 2nd previous post, I re-read Peter Williams post, linked via the same debate. I recall reading this in 2005.

Martin Fowler has posted a nice article on humane interface design (as opposed to minimal interface design). I am definitely on the side of right and good (read: humane interfaces) in this debate. Nothing takes the fun out of programming faster than having to write a bit of code that you know ...

Read More Source

September 7th, 2017

In Technology

No Comments

The failure of the X Window system paved the road for the rise of the Web

This is an amazing article about the absolute failure of the X Window system. Copyright (C) 1994, so this was written just as the Web was gaining momentum, but before it was obvious that the Web would become the main way that different devices could establish a single interface. Among the many good bits, there is this:

Figuring out where a particular resource value is for a running application is much fun, as resource can have come from any of ...

Read More Source

September 7th, 2017

In Technology

No Comments

The loneliness of advanced skills

I decided to re-read The departure of the hyper-enthusiasts. I’ve linked to that essay many times before. It was formative for me, at a formative moment for the Web, so it was formative in many ways, and it captured something important about that moment, and how things were changing.

I find myself getting nostalgic. That is not interesting in and of itself. To be nostalgic for a formative time, about a formative essay, is ordinary enough.

But I had an ...

Read More Source

September 7th, 2017

In Technology

No Comments

Blue or Not Blue decision trees

This is a point worth remembering:

“Before we get started I need to clarify something. Theoretical decision trees can have two or more branches protruding from a single node. However, this can be computationally expensive so most implementations of decision trees only allow binary splits.”

Post external references 1https://gormanalysis.com/magic-behind-constructing-a-decision-tree/ Source

September 7th, 2017

In Technology

No Comments

If NodeJS is so great, why are the problems so common?

Callback hell. Take a deep breath and ask yourself why this website needs to exist:

Post external references 1http://callbackhell.com/ Source

August 12th, 2017

In Technology

No Comments

Supervisord has spent 5 years discussing whether or not it should support a timing mechanism?

I’m not sure what to make of this incredible thread in the issue tracker on Github for the Supervisord repo. Someone asked, in 2012, for a way to time when various apps start, so that they could start app A, B and C in order. On April 16 of this year, 2017, someone wrote:

This issue has been open since 2012, does anyone have ownership over it? It’s clearly important to many users; is there any consensus amongst supervisor contributors?

I ...

Read More Source

August 4th, 2017

In Technology

No Comments

A pure GUI for IP/TCP, a technology with a single mandate, with no burden of also offering semantics or structure or hierarchy

Someone on Hacker News said most developers don’t “get” the Web, and I posted this in response:

My history is surprisingly similar to yours, I started in 1999, I used Notepad as my first text editor, and by 2003 I got caught up in the movement towards making markup strict, which I felt was the mark of professionalism. However, by 2006 I had mostly rejected the notion of “strictness”. There were several things that turned me against strictness. One of ...

Read More Source

August 2nd, 2017

In Technology

No Comments

Cultural differences between Clojure and NodeJS

Right now I have to work with NodeJS, because I have a client who uses NodeJS. I am getting used to the asynchronous style of NodeJS. I notice a subtle cultural difference between Clojure and NodeJS. Most of the tutorials for NodeJS assume the person reading is a beginner, someone who may not understand big complex words like “asynchronous”. In the world of Clojure, there is the assumption that the person reading the article has some experience. With Clojure, there ...

Read More Source

July 24th, 2017

In Technology

No Comments

NodeJS remains single threaded

This is a good example of the limits of NodeJS:

async.parallel([ function(callback){ setTimeout(function(){ callback(null, 'one'); }, 200); }, function(callback){ setTimeout(function(){ callback(null, 'two'); ...
Read More Source

July 19th, 2017

In Technology

No Comments

No one should ever use NodeJS, part MMCVXII

I previously wrote about a very surprising bug I discovered in NodeJS/HapiJS.

But today I face a new issue: How to parallelize work in NodeJs? I discovered this fantastic library called paralleljs. It assigns work to child processes. Fantastic idea, very similar to Python. And how do I get required libs into the child process? That is very difficult. From the issues on Github:

to make things a bit more concrete, this snippet would be representative of what I want ...

Read More Source

July 18th, 2017

In Technology

No Comments

Do people love DigitalOne servers?

I’m looking to host my next web site at DigitalOne. If you have experience with them, please let me know.

Post external references 1http://www.digitalone.com/#!server=1&memory=13&firstraid=73&hdd1_2=228&port=17&ip=20&os=71&panel=72&billingcycle=/step1 Source

July 18th, 2017

In Technology

No Comments

When Nginx becomes the bottleneck

This makes me sad. I love Nginx as a reverse proxy, so long as it is invisible and I never have to think about. Realizing that it, too, can be a source of problems really is discouraging.

47,135 connections in TIME_WAIT! Moreover, ss indicates that they are all closed connections. This suggests the server is burning through a large portion of the available port range, which implies that it is allocating a new port for each connection it’s handling. Tweaking ...

Read More Source

July 18th, 2017

In Technology

No Comments

Trying to explicitly enumerate everything that is true is hopeless

This is good:

Let’s take a look at some existing representations. The most famous representation is WordNet. In WordNet, the symbols are groups of words that have the same meaning, called synsets. One synset could be the set consisting of “car” and “automobile.” Each word can be in multiple synsets. For example, “bank” could be in the synset that means river bank and also in the synset that means a place where money is deposited. There are a few kinds ...

Read More Source

July 8th, 2017

In Technology

No Comments

A simple intro to tokenizing with OpenNLP in Clojure

A nice intro:

Finding sentences

Words (tokens) aren’t the only structures that we’re interested in, however. Another interesting and useful grammatical structure is the sentence. In this recipe, we’ll use a process similar to the one we used in the previous recipe, Tokenizing text, in order to create a function that will pull sentences from a string in the same way that tokenize pulled tokens from a string in the last recipe.

Getting ready

We’ll need to include clojure-opennlp in our project.clj file:

(defproject com.ericrochester/text-data ...

Read More Source

July 8th, 2017

In Technology

No Comments

Neural Networks beat Support Vector Machine beat Random Forests

I’ve written about my experience at a startup that used NLP to translate a salesperson’s text message into an entry in Salesforce. At the time, our NLP developer used the Stanford NLP library to try to build a model, but mostly they used a lot of regex and string matching.

I’ve been thinking about how we might have done that project faster and better. I’ve recently been thinking the right approach would have been Random Forests. We were lucky ...

Read More Source

July 8th, 2017

In Technology

No Comments

The struggle to find meaning in words, via word counts in matrixes

Interesting comment:

With LSA, each document is transformed into a single vector that has the length of the vocabulary. The length of the vocabulary is the number of unique words across all documents. If a word is present in a document, it is represented as a 1 in the vector and 0 if it is not. So after this transformation, the text is transformed in an D by V matrix where D is the number of documents and V is the ...

Read More Source

July 8th, 2017

In Technology

No Comments

The problems with DynamoDB

Everything I’ve read suggests to me that companies are drifting towards Cassandra as the best of the NoSQL databases. But some of my clients give me directives such as “Please restrict yourself to AWS technologies”. So I was considering using DynamoDB. And then I read this.

Interesting:

When a customer does an action, it is reasonable to think that this kicks off a number of actions in your system. One action might spawn a number of events to be sent into ...

Read More Source

June 29th, 2017

In Technology

No Comments

The curse of too many dimensions

Damn, this is interesting:

The curse of dimensionality refers to various phenomena that arise when analyzing and organizing data in high-dimensional spaces (often with hundreds or thousands of dimensions) that do not occur in low-dimensional settings such as the three-dimensional physical space of everyday experience. The expression was coined by Richard E. Bellman when considering problems in dynamic optimization.

There are multiple phenomena referred to by this name in domains such as numerical analysis, sampling, combinatorics, machine learning, data mining, and ...

Read More Source

June 29th, 2017

In Technology

No Comments

Morphism is the starting point of category theory

I guess I knew this, but still it is interesting to read:

In algebra, a homomorphism is a structure-preserving map between two algebraic structures of the same type (such as two groups, two rings, or two vector spaces). The word homomorphism comes from the ancient Greek language: ὁμός (homos) meaning “same” and μορφή (morphe) meaning “form” or “shape”.

Homomorphisms of vector spaces are also called linear maps, and their study is the object of linear algebra.

The concept of homomorphism has been ...

Read More Source

June 29th, 2017

In Technology

No Comments

All the weird characters in Clojure explained

This is a very useful guide.

Post external references 1https://clojure.org/guides/weird_characters Source

May 31st, 2017

In Technology

No Comments

Cost cutting at British Airways caused the catastrophic failure of its database service

This is obviously a lie:

BA insisted that the computer system outages that grounded hundreds of flights were not linked to ferocious cost-cutting and the outsourcing of work to contractors in its IT department.

There is absolutely no way to get a melt down of this scale unless you’ve just fired your key staff, so you’ve nobody left who knows how to bring the system back online. I’ve worked at very small startups, with just 3 people on staff, and we ...

Read More Source

May 27th, 2017

In Technology

No Comments

Is Clojure dying?

Interesting.

The past week or so a lot of discussion and introspection has been happening in the Clojure community. Eric Normand responded to my one year Lambda Island post with some reflections on the size and growth of the community.

And then Zack Maril lamented on Twitter: “I’m calling it, clojure’s dying more than it is growing”. This sparked a mega-thread, which was still raging four days later. A parallel discussion thread formed on Reddit. Someone asked if their were any Clojure ...

Read More Source

May 9th, 2017

In Technology

No Comments

Don’t use PM2 with NodeJS

Interesting:

I don’t use any of the “node” process managers because they don’t solve the problem of starting your process at server boot. I use init scripts, or upstart, or runit or systemd like you’re supposed to for a service. Anyone using forever or pm2 likely doesn’t understand sysops.

Post external references 1https://groups.google.com/forum/#!searchin/nodejs/pm2%7Csort:relevance/nodejs/5kb1QAuGxcc/Dj6jMpmimRwJ Source

May 5th, 2017

In Technology

No Comments

Very long GC pauses when NodeJS is about to die from memory exhaustion

I sometimes see stuff like this in my logs, when my NodeJS app has reached critical:

856870 ms: Mark-sweep 5647.1 (6029.8) -> 5625.8 (6029.8) MB, 10733.9 / 0.0 ms [allocation failure] [GC in old space requested]. 867774 ms: Mark-sweep 5625.8 (6029.8) -> 5625.8 (6030.8) MB, 10903.4 / 0.0 ms [allocation failure] [GC in old space requested]. 878580 ms: Mark-sweep 5625.8 (6030.8) -> 5625.8 (5996.8) MB, 10806.3 / 0.0 ms [last resort gc]. 889353 ms: Mark-sweep ...
Read More Source

April 26th, 2017

In Technology

No Comments

More NLP for Clojure

This looks fantastic:

To do its magic, postagga extracts the phrase structure of your input, and tries to find how do this structure compare to its many semantic rules and if it finds a match, where in this structure shall he extract meaningful information.

Let’s study a simple example. Look at the next sentence:

“Rafik loves apples” That is our “Natural language input”

First step in understanding this sentence is to extract some structure from it so it is easier to interpret. One common way ...

Read More Source

April 26th, 2017

In Technology

No Comments

TCP can fail in many, many ways

Amazing that the world depends on a technology with so many quirks:

So, if we read that data first, and LINGER, are we good to go? Not really. The close() call really does not convey what we are trying to tell the kernel: please close the connection after sending all the data I submitted through write().

Luckily, the system call shutdown() is available, which tells the kernel exactly this. However, it alone is not enough. When shutdown() returns, we still have no indication ...

Read More Source

April 23rd, 2017

In Technology

No Comments

Thinking about Concurrency, Raymond Hettinger, Python core developer

A basic overview of concurrency, most of which I knew, but he had one good line at the end: “If you add enough locks to your concurrent code, it is no longer concurrent. It is completely sequential, you’ve simply made it a lot more complicated than single threaded code.”

Post external references 1https://www.youtube.com/watch?v=Bv25Dwe84g0 Source

April 13th, 2017

In Technology

No Comments

Python has greenlets

I’m sad that Python is doing so well. It gets all the attention that should go to Clojure. They are both great for AI and NLP, but Python doesn’t have as many great ideas for concurrency. Or rather, the language doesn’t, but the eco-system makes up for that with a wealth of interesting ideas — and it is sad to see so much effort go to see making Python work, rather than just starting over with something that has a ...

Read More Source

April 13th, 2017

In Technology

No Comments

Katharine Jarmul describes the bias in Google’s Word2Vec software

Interesting:

I must warn you that parts of this post are disgusting, disturbing and awful. If you are having a rough day, feel free to save for another time. If you are already sick of seeing hateful language, this is likely not a post to read at present. That said, I feel my duty as a former journalist to look at it, expose it, and hope to spark better conversations around how we handle both implicit and explicit bias and prejudice ...

Read More Source

April 11th, 2017

In Technology

No Comments

Why are people ignoring the problems with NodeJS?

Two weeks ago I wrote “A surprising NodeJS failure mode: deterministic code becomes probabilistic under load“. Since then I’ve been reading up on NodeJS and learning more about its substantial failure modes. I’m left feeling very surprised at the success that NodeJS is having.

Just to give you a sense of what I mean, this is how The New Stack sums up the success of NodeJS:

Ready for a Long Term Node Relationship?

In just seven short years, Node.js has gone ...

Read More Source

March 30th, 2017

In Technology

No Comments

A surprising NodeJS failure mode: deterministic code becomes probabilistic under load

I have a client. Let’s say they run some scripts that scrape the web for data, then they run some NLP scripts to pull out the facts they need, then they need to insert that into their MySQL database. When they gather facts, they need to know if the facts are about an organization that is already in the database. So they asked me to create an API that could take their new facts, and try to match those facts ...

Read More Source

March 29th, 2017

In Technology

No Comments

Skip Lists: A Probabilistic Alternative to Balanced Trees

Interesting:

Skip lists are a data structure that can be used in place of balanced trees. Skip lists use probabilistic balancing rather than strictly enforced balancing and as a result the algorithms for insertion and deletion in skip lists are much simpler and significantly faster than equivalent algorithms for balanced trees

A node that has k forward pointers is called a level k node. If every (2i)th node has a pointer 2i nodes ahead, then levels of nodes are distributed in a simple pattern: 50% are level 1, 25% ...

Read More Source

March 29th, 2017

In Technology

No Comments

Novelty seeking multi agent systems

Interesting:

This paper considers novelty-seeking multi-agent systems as a step towards more efficient generation of creative artifacts. We describe a simple multi-agent architecture where agents have limited resources and exercise self-criticism, veto power and voting to collectively regulate which artifacts are selected to the domain i.e., the cultural storage of the system. To overcome their individual resource limitations, agents have a limited access to the artifacts already in the domain which they can use to guide their search for novel artifacts. Creating geometric images called spirographs as a case study, we show ...

Read More Source

March 29th, 2017

In Technology

No Comments

Bushy join trees for snowstorm queries

I am taken aback by the amount of jargon here:

Many workloads for analytical processing in commercial RDBMSs are dominated by snowstorm queries, which are characterized by references to multiple large fact tables and their associated smaller dimension tables. This paper describes a technique for bushy join tree optimization for snowstorm queries in Oracle database system. This technique generates bushy join trees containing subtrees that produce substantially reduced sets of rows and, therefore, their joins with other subtrees are generally much ...

Read More Source

March 29th, 2017

In Technology

No Comments

How to combine novelty search with fitness-based evolution

I need to go back and read this whole article:

Novelty search is a state-of-the-art evolutionary approach that promotes behavioural novelty instead of pursuing a static objective. Along with a large number of successful applications, many different variants of novelty search have been proposed. It is still unclear, however, how some key parameters and algorithmic components influence the evolutionary dynamics and performance of novelty search. In this paper, we conduct a comprehensive empirical study focused on novelty search’s algorithmic components. We ...

Read More Source

March 29th, 2017

In Technology

No Comments

Novelty search

Interesting:

Novelty search is a recent algorithm geared toward exploring search spaces without regard to objectives. When the presence of constraints divides a search space into feasible space and infeasible space, interesting implications arise regarding how novelty search explores such spaces. This paper elaborates on the problem of constrained novelty search and proposes two novelty search algorithms which search within both the feasible and the infeasible space. Inspired by the FI-2pop genetic algorithm, both algorithms maintain and evolve two separate populations, ...

Read More Source

March 27th, 2017

In Technology

No Comments

What every object on a network needs

There is no need to rely on the Web for these things:

The problem with operations is that we have to define, a-priori, the semantics of each individual operation. We cannot tell, unless we know beforehand, whether the operation is safe to call multiple times, whether the result of an operation will be the same every time we call it, whether the result can be cached, and if so for how long. Many years of building distributed applications have told ...

Read More Source

March 27th, 2017

In Technology

No Comments

Use pseudo URIs for an id

A long article, but this is interesting:

When facing the problems above, my team at SoundCloud started exploring alternatives that would allow for us to have simple, scalar values that were still rich enough to act as good identifiers across our hundreds of microservices. Reading through decades of industry work on the matter, we found something simple that could help us: Uniform Resource Names, or URNs. URNs were a type of Uniform Resource Identifiers (URIs) that, as opposed to URLs, were ...

Read More Source

March 27th, 2017

In Technology

No Comments

Is NewSQL the next big thing?

I like the fact that Kinesis has an SQL interface:

I can see this being useful for Business Intelligence. But that seems like a niche to me. Both SQL and NoSQL solved big, universal problems.

But maybe the SQL is just window dressing? Maybe the real breakthrough is a distributed system with strong consistency guarantees? It would be a very big deal if someone found a way around the CAP Theorem. But otherwise, NewSQL is just more bad marketing for ...

Read More Source

March 27th, 2017

In Technology

No Comments

Designing a neural net is really hard

Interesting:

2bitencryption says:

Designing a neural network is a thousand times harder than I imagined.

After AlphaGo, I tasked myself with creating a neural network that would use Q-Learning to play Reversi (aka Othello).

At that point, I had already utilized Q-Learning (the tabular version, not using a neural network) for some very simple and mostly proof-of-concept projects, so I understood how it worked. I read up only perceptrons, relu, the benefits/disadvantages of having more/fewer layers, etc.

Then I actually started on the project, thinking ...

Read More Source

March 26th, 2017

In Technology

No Comments

Training a neural net

A great intro:

Training

Training is how we teach a neural network what we want it to learn. It follows a simple five step process:

1.) Create a training data set, which we will call x and load its labels as targets y

2.) Feed the x data forward through the network with the result being predictions y’

3.) Figure out the “loss” of the network, which is the difference between the predictions y’ and the correct targets y

4.) Compute the “gradient” of the loss ...

Read More Source

March 26th, 2017

In Technology

No Comments

Transform real data into a tensor, that is, tensor flow

A good introduction:

Post external references 1https://hackernoon.com/learning-ai-if-you-suck-at-math-p4-tensors-illustrated-with-cats-27f0002c9b32#.ss8u15jah Source

March 17th, 2017

In Technology

No Comments

What if we had computer bits that allowed more than two values?

The Red-Black tree is interesting because it can store the color using only bit, and one bit can have one of two values, often represented as either a 0 or a 1. That is the nature of our silicon hardware. But if we had hardware where the basic charge could be one of several values, then it would possible to do multi colored trees. If we had hardware that could hold one of 20 values, it would be possible to ...

Read More Source

March 17th, 2017

In Technology

No Comments

An InputStream requires more memory as it has to buffer the whole file

This is exactly the opposite of what I’ve always understood, and what I’ve read elsewhere:

When opening a workbook, either a .xls HSSFWorkbook, or a .xlsx XSSFWorkbook, the Workbook can be loaded from either a File or an InputStream. Using a File object allows for lower memory consumption, while an InputStream requires more memory as it has to buffer the whole file.

The stream requires more memory? That is crazy. It’s a stream. And what is a File object if not ...

Read More Source

March 8th, 2017

In Technology

No Comments

Joe Armstrong: computer programming is the alteration of values in a key value database

An incredible assertion that would be dismissed if it was coming from someone less talented than Joe Armstrong. It is interesting to think that RAM is a key value store, and so anything that updates RAM is altering the values of a key value store.

He starts by saying that he often ends up with miscellaneous code:

When I’m programming I often get to the point were I say there shoulda function foo/2 in lists.erl but their isn’t. There should be ...

Read More Source

March 1st, 2017

In Technology

No Comments

How to get Nginx to wait for your app

I’ve been struggling with a slow NodeJS app which is behind a reverse proxy handled by Nginx.

I found this bit useful:

Syntax: proxy_max_temp_file_size size; Default: proxy_max_temp_file_size 1024m; Context: http, server, location When buffering of responses from the proxied server is enabled, and the whole response does not fit into the buffers set by the proxy_buffer_size and proxy_buffers directives, a part of the response can be saved to a temporary file. This directive sets the maximum size of the temporary file. The size of data written to ...
Read More Source

February 28th, 2017

In Technology

No Comments

How do we teach self-driving cars to avoid hitting people on bicycles?

This is a good conversation:

ChuckMcM :

“Now Johnny, when you ride your bike you must wear your I-am-a-bike vest and follow these patterns or the cars are likely to kill you.” :-) Summary is that “people riding bikes” (PRB) is a much denser image set than “people in cars” (PIC) and “people walking or jogging” (PWJ), and the PRB objects have a much higher dynamic angular vector capability (they can change direction extremely quickly) combined with a wider dynamic velocity vector ...

Read More Source

February 14th, 2017

In Technology

No Comments

Different NLP approaches

A nice summary:

With LSA, each document is transformed into a single vector that has the length of the vocabulary. The length of the vocabulary is the number of unique words across all documents. If a word is present in a document, it is represented as a 1 in the vector and 0 if it is not. So after this transformation, the text is transformed in an D by V matrix where D is the number of documents and V is ...

Read More Source

February 11th, 2017

In Technology

No Comments

Tony Arcieri’s very good defense of Rails

I just had a project where I had to write an app that could fire several million requests against an API, with data pulled from a database. I wrote the whole thing in NodeJS and HapiJS, and it was an agony. The uncontrolled async of NodeJS tripped me up. If I need to fire 10 million HTTP requests, how do I do that without blowing the stack? As it was, the functions would pile up, and then I’d get a ...

Read More Source

February 7th, 2017

In Technology

No Comments

systemd is no longer interested in the grubby details of reality

Interesting:

Unfortunately, since then systemd developers have shown an unfortunate and increasing streak of idealism. More and more, systemd seems not to be interested in dealing with the world as it actually is, with all of its grubby inconvenient mess; instead it simply assumes some pure and ideal version of things. If the world does not measure up to how it is supposed to be, well, that is not systemd’s problem. Systemd will do the nominally right thing no matter how ...

Read More Source

January 28th, 2017

In Technology

No Comments

NLP as a simple four-step formula: embed, encode, attend, predict

Interesting:

Let’s say you have a great technique for predicting a class ID from a dense vector of real values. You’re convinced you can solve any problem that has that particular input/output “shape”. Separately, you also have a great technique for predicting a single vector from a vector and a matrix. Now you have the solutions to three problems, not two. If you start with a matrix and a vector, and you need a class ID, you can obviously compose the ...

Read More Source

January 27th, 2017

In Technology

No Comments

Paul Phillips denounces Scala

Pacific Northwest Scala 2013 We’re Doing It All Wrong by Paul Phillips For a computer programming tech conference, this talk has a surprising amount of emotional rawness. Phillips is denouncing Scala both on a personal level and on a technical level.

Post external references 1https://www.youtube.com/watch?v=TS1lpKBMkgg Source

January 27th, 2017

In Technology

No Comments

One team that gave up on Scala

It’s interesting how Scala was initially seen as “A Better Java” but now is seen as having been to experimental in its approach.

I remember when I first saw the potential issues of scaling Scala at Gravity back in 2009/10ish. It was close to the end of the day when we had a major issue reported in production that was affecting our large customers. Several of us started investigating and were able to track the source of the issue. The ...

Read More Source

January 27th, 2017

In Technology

No Comments

Scala programmers do not trust Martin Odersky

This whole conversation is incredible. Here is a forum full of Scala programmers, and they seem to be talking against Odersky’s proposal. And with some anger. I’ve never seen a Rubyist talk about Matz like that. I’ve never heard a Clojurist talk about Rich Hickey that way. Java programmers are, at most, muted in their criticism of Gosling, with the overall recognition that “mistakes were made” but made in good faith. I suppose you could argue that Pythonists are split ...

Read More Source

January 27th, 2017

In Technology

No Comments

The failure of hierarchy based on category

This is good:

The Hierarchy Problem

Every time I start at a new company, I struggle with the problem when I’m creating a place to put my Company Documents, e.g. the Employee Handbook.

Do I create a folder called Documents and then create a folder called Company in that?

Or do I create a folder called Company and then create a folder called Documents in that?

Both work. But which is right? Which is best?

The idea of Categorical Hierarchies was that there were Base Classes ...

Read More Source

January 27th, 2017

In Technology

No Comments

Logicians hate randomness

My strongest subject was always logic, even though I was terrible at most other forms of math. And though I have doubts about randomness, I certainly love inconsistency, which can be approached in a logical way. Every computer programmer who writes concurrent code has to deal with inconsistent data, and it would be great if logicians thought more deeply about how to formalize this.

With Gödel, at first there was a lot of shock. As a student in the late ...

Read More Source

January 24th, 2017

In Technology

No Comments

Movio.co decides to switch from Scala to Go, and they reference my essay

Since they cite me (and my essay from 2014) as part of their decision making process, I want to throw in 2 cents here. They ended up deciding on Go, whereas I have ended up preferring Clojure, yet I agree with a lot of what they say, so I’ll try to clarify why I ended up with a different decision than what they made.

I understand what they mean when they write:

I think the first time I appreciated the positive ...

Read More Source

January 18th, 2017

In Technology

No Comments

Modern programming depends on DSLs, and Lisp is the best as DSLs

Anders Hovmöller seems to be stuck with a mentality shaped by the C++ language, and so he doesn’t get the important of DSLs.

In C++ land it’s well known that you must keep to a subset of the language in a code base to keep sane. It’s also known that every time you increase the size of the subset you are making the situation worse even if it solves a small problem at hand.

In Lisp land on the other hand, all ...

Read More Source

January 8th, 2017

In Technology

No Comments

This year’s goal: move away from WordPress

Like a lot of people, I first used WordPress because it was quick and easy to get going, and because my designer knew it very well. But I dislike the dependence on MySQL, I dislike its slowness, and I dislike the security problems that I’ve had (this site has been hacked twice).

For my modest goals, I only need a static site, so I plan to switch to Cryogen. I was interested by this article:

For a while I ...

Read More Source

January 8th, 2017

In Technology

No Comments

Different people have different strengths

I love this :

I’ve had similar self-doubt in the past, I considered myself fast but sloppy. I changed my opinion after a workshop on different team roles at my previous job. A lot of it was boring workshop fluff, but I loved the core message: that many personality traits aren’t purely positive or negative. Perfectionists are nitpickers. Fast developers are sloppy. Experienced ones overthink stuff. Bleeding-edge evangelists ruin long-term stability etc. There are two sides to every coin. We tend to ...

Read More Source

January 8th, 2017

In Technology

No Comments

Can event logs, as a software architecture, really work?

This is an interesting bit of criticism of this architecture, an architecture which certainly has gotten a lot of attention over the last 4 years:

I have worked on, or cleaned up, 4 different CQRS/ES projects. They have all failed. Each time the people leading the project and championing the architecture were smart, capable, technically adept folks, but they couldn’t make it work.

There’s more than one flavor of this particular arch, but Event Sourcing in general is simply not very ...

Read More Source

December 15th, 2016

In Technology

No Comments

Clojure / Java app dying while using XSSF and Sax parsers to import Excel files

I posted this question to StackOverflow and I would appreciate any help figuring out what the memory issue is:

Clojure / Java app dying while using XSSF and Sax parsers to import Excel files

Post external references 1http://stackoverflow.com/questions/41169687/clojure-java-app-dying-while-using-xssf-and-sax-parsers-to-import-excel-files Source

December 14th, 2016

In Technology

No Comments

Ugly engineering

This sounds hideous:

I used to work on the DynamoDB team. Throwaway account because my normal account can be tied back to my real name.

“Each hash key resolves to a number of possible servers the data can be on. Data is replicated across several of these servers. For redundancy. The hash key determines which shard to use. On individual machines, each set of data is stored by a compound key of hash key and sort key (if there is a sort ...

Read More Source

December 13th, 2016

In Technology

No Comments

Beautiful engineering

Wow. Sometimes I witness some engineering that leaves me feeling joy and inspiration. I felt that way about the Golden Gate Bridge. But sometimes I feel that way about code. For instance, right now, I just learned that the defun macro for Clojure allows this function for calculating a fibonacci number.

A fibonacci function: (defun fib ([0] 0) ([1] 1) ([n] (+ (fib (- n 1)) (fib (- n 2))))) Output: (fib 10) ;; 55

That ...

Read More Source

December 7th, 2016

In Technology

No Comments

The crazy process of trying to release a Salesforce app

Right now I’m helping a client release an app for Salesforce in the Salesforce App Exchange. The process for releasing an app is completely insane and very poorly documented (or rather, there is an abundance of documentation for various parts of the process, but there is no summary of the process). I was lucky to stumble upon this post, which offers the nearest thing to a summary of the process:

1) Create a Dev Org

2) Become a Salesforce.com Partner

3) Create Test ...

Read More Source

November 26th, 2016

In Technology

No Comments

Clojure for XML

This is a fantastic overview of different approaches:

Zippers are probably the easiest way to manage xml – once you grok them.

Zippers are a strange beast. Wikipedia describes them as:

A technique of representing an aggregate data structure so that it is convenient for writing programs that traverse the structure arbitrarily and update it’s contents…

I like to think of a zipper as a kind of pointer to part of a tree – at any time if you have a tree of nodes ...

Read More Source

November 26th, 2016

In Technology

No Comments

How far can code be optimized

This is funny and interesting:

When you’re competing with another player, they will probably find a way to beat your score by just a few points. Let’s say my score is 340 and a friend beats me with a score of 335. (lower is better. The score is just the number of executed instructions) What follows is a bunch of head-scratching about how you could possibly get any more cycles out of the algorithm. After an hour of staring and trying ...

Read More Source

November 14th, 2016

In Technology

No Comments

NLP with Clojure and OpenNLP

This is a fantastic introduction:

Natural Language Processing (NPL) opens the door to the possibility of turning otherwise inert text into meaningful or, more interestingly, actionable information. It is the latter that I am interested in and what this installment will focus on. I will explore the basics of NLP using the OpenNLP library and Clojure to convert a sentence into a useful structure to store or act on. More specifically, my goal is to take simple sentences that indicate the ...

Read More Source

November 13th, 2016

In Technology

No Comments

The best interview technique: pay people $200 to attempt some task over the weekend

I like this idea:

In regards to what works the best, I found that these 2 ideas work the best when combined.

PAID Sample project assignment (err on the side of paying fairly — say $100+/hour for estimated completion time — if the problem should require 2 hours to complete, offer $200)

Bring the candidate in and discuss the solution. Let the candidate talk about their design decisions, challenge them as you would any team member and let them provide their reasoning.

Paying candidates to work on a ...

Read More Source

November 5th, 2016

In Technology

No Comments

The perfect Javascript file uploader?

This looks amazing:

The default behavior of Fine Uploader is to immediately attempt to upload files as they are selected. One option allows you to simply queue all files, and then start uploading at a later time by calling uploadStoredFiles() on your Fine Uploader instance. You can also easily allow your users to edit the names of each submitted file before uploading.

This Fine Uploader instance below also demonstrates the edit file name feature, which allows you to edit the name of ...

Read More Source

October 31st, 2016

In Technology

No Comments

Stripe almost got burned by Bitcoin

It’s amazing how many disasters seem to happen whenever companies try to use crypto-currency:

For a variety of reasons, it is sometimes necessary to refund bitcoin transactions: For example, a customer cancelling their order; accidentally sending in the wrong number of bitcoins; or even sending in the correct number of bitcoins, but not within the requisite time window, resulting in their value being lower than necessary. Consequently, Stripe allows for bitcoin transactions to be refunded — with the caveat that, for ...

Read More Source

October 27th, 2016

In Technology

No Comments

Software doesn’t need options

At least for the consumer. An API for a developer is something else completely.

Great software is:

Opinionated Options are a cop out. It’s the designer’s job to figure out the best way to use the product, and delegating that choice to the user is pure laziness. Hive only resorts to options as a last resort.

Post external references 1https://blog.hive.com/hives-7-design-principles-c59d593ae58b#.wfcfdqavm Source

October 3rd, 2016

In Technology

No Comments

How process becomes bloated and bureaucratic over time

This is good:

Checkout/Ordering team: “Your Payment Processing release broke Checkout in production.”

Payment Processing team: “Didn’t know this would affect Checkout. Had no time to look into it because we were too busy working on the new payment functionality in the iOS app.”

Mobile App Product Visionary: “I want my own dedicated team. This happened because Payment Processing is spread too thin to work on the app and the backend at the same time.”

Development Manager: “Org chart change – let’s have all ...

Read More Source

September 18th, 2016

In Technology

No Comments

Rapidly Exploring Random Trees

Interesting. It says the RRT proceeds by breaking the Voronoi spaces into smaller Voronoi spaces. The algorithm has a bias towards unexplored territory, unlike a purely random algorithm, which tends to loop back to explored space. I didn’t dig into the algorithm, but I can imagine how this would be useful in some situations.

Post external references 1http://msl.cs.uiuc.edu/~lavalle/papers/Lav98c.pdf Source

September 18th, 2016

In Technology

No Comments

As a consolidation client for Kinesis, ZeroMQ is much better than Kafka

This is fantastic:

With the problem of real-time log consolidation solved with ZeroMQ, supporting durable storage of logs for later post-processing becomes a smaller and a much more manageable challenge. For example, off the shelve solutions like Logstash are capable of capturing data from ZeroMQ and publishing it to a variety of destinations. In our particular situation at Auth0, we are already using AWS Kinesis, ElasticSearch, and Kibana as a log processing pipeline in other parts of our operations. As such we ...

Read More Source

September 18th, 2016

In Technology

No Comments

I always learn what I need to know about a year too late

At Celolot I wrote a bunch of Clojure apps that talked to each other over Redis thanks to Carmine/Nippy. We passed around maps, and each app checked a field in the map to see if the map was a message that it was suppose to respond to.

But I bet this is a thousand times faster:

ZeroMQ does not have a first class notion of a topic, yet it does have a first class concept of a subscription filter. Subscription filters ...

Read More Source

September 18th, 2016

In Technology

No Comments

Camille Fournier on ephemeral nodes in etcd

I missed this the first time. In etcd a service needs to constantly update the fact that it is alive. If it dies, its info disappears. So other services can contact etcd and ask “Who is alive right now?”

ZooKeeper allows stateful connections. Thick clients. But what happens when you try to use ZooKeeper from a language that does not allow threads? Do you block on the only thread that you have?

For pub-sub she recommends RabbitMQ or ZeroMQ.

Post external references 1http://www.ustream.tv/recorded/61483409 Source

September 18th, 2016

In Technology

No Comments

Why would I ever want to use Docker?

I’m trying to be open minded about Docker, but it is a struggle to grok why this is good. Do I want to bind all dependencies to my app? Great, then I’ll build uberjars that combine everything into 1 binary. But how will I work with Ruby On Rails? That’s easy: it is time to move away from Ruby On Rails. If Ruby can only be kept alive by using Docker, then we should give up on Ruby.

How will ...

Read More Source

September 18th, 2016

In Technology

No Comments

Unary Numeral System

Interesting:

Addition and subtraction are particularly simple in the unary system, as they involve little more than string concatenation. The Hamming weight or population count operation that counts the number of nonzero bits in a sequence of binary values may also be interpreted as a conversion from unary to binary numbers. However, multiplication is more cumbersome and has often been used as a test case for the design of Turing machines.

Compared to standard positional numeral systems, the unary system is inconvenient ...

Read More Source

September 18th, 2016

In Technology

No Comments

It took me 10 years to write that 5 line app

There is the story where Picasso paints a quick sketch and wants a huge amount of money for it, and the art collector says “It only took you 15 minutes to make that sketch!” and Picasso says “I spent 60 years learning how to do that in 15 minutes.”

Maybe something like this applies when it comes to microservices. When I advocate for microservices, I bring 17 years of experience to the conversation. The first 7 of those years meant ...

Read More Source

September 18th, 2016

In Technology

No Comments

For the last 10,000 years, the banking industry has relied on eventual consistency

Why do computer programmers feel smug offering the following example, when it is so clearly wrong?

Transactions, to a database, are important, because banks must keep track of money. Suppose a person were to move $100 from their Savings account to their Checking account. Suppose $100 is added to checking, and just then the electricity dies, and the computers die, before $100 can be subtracted from their Savings account. The person now has an extra $100, which they should not ...

Read More Source

September 17th, 2016

In Technology

No Comments

I keep really good company

This article by Yegor Bugayenko cites the following people:

Edsger W. Dijkstra

Alan Kay

Paul Graham

Richard Mansfield

Eric Raymond

Jeff Atwood

Linus Torvalds

Oscar Nierstrasz

Rich Hickey

Eric Allman

Joe Armstrong

Rob Pike

John Barker

Lawrence Krubner (me)

Asaf Shelly

I must be doing something right to get cited alongside Edsger W. Dijkstra, Alan Kay, Paul Graham, Linus Torvalds, and Rich Hickey.

Post external references 1http://www.yegor256.com/2016/08/15/what-is-wrong-object-oriented-programming.html Source

September 17th, 2016

In Technology

No Comments

When is consistency needed in a database?

This is a great comment :

Consistently is overvalued. Requiring consistency in distributed system generally leads to designs that reduces availability. Which is one of the reasons that bank transactions generally do not rely on transactional updates against your bank. “Low level” operations as part of settlement may us transactions, but the bank system is “designed” (more like it has grown by accretion) to function almost entirely by settlement and reconciliation rather than holding onto any notion of consistency. The real world rarely involves ...

Read More Source

September 11th, 2016

In Technology

No Comments

Why did Salesforce use markup tags for VisualForce?

By now, I think we can agree, using HTML for the web was a mistake. But Sir Tim Berners-Lee could not have known that back in 1989. He was thinking that he could create a Semantic Web, and for that, the use of a markup language was defensible. Only in retrospect do we see that HTML was mostly used as a GUI for TCP/IP. Eventually Sir Tim Berners-Lee conceded the point, and so he went off and created RDF.

But ...

Read More Source

September 5th, 2016

In Technology

No Comments

Comparing data sets that are imbalanced

Interesting:

Research on imbalanced classes often considers imbalanced to mean a minority class of 10% to 20%. In reality, datasets can get far more imbalanced than this. —Here are some examples:

1.) About 2% of credit card accounts are defrauded per year1. (Most fraud detection domains are heavily imbalanced.)

2.) Medical screening for a condition is usually performed on a large population of people without the condition, to detect a small minority with it (e.g., HIV prevalence in the USA is ~0.4%).

3.) Disk ...

Read More Source

September 5th, 2016

In Technology

No Comments

Base 10 numbers need log(10) digits to be described

I’m sure I knew this example once, but I’d completely forgotten it, and it is so perfectly obvious when we talk to non-technical people and they ask for an example of what logarithmic growth looks like:

In mathematics, logarithmic growth describes a phenomenon whose size or cost can be described as a logarithm function of some input. e.g. y = C log (x). Note that any logarithm base can be used, since one can be converted to another by multiplying ...

Read More Source

September 5th, 2016

In Technology

No Comments

Accessing any single element in an array takes constant time as only one operation has to be performed to locate it.

This seems like it could be used as a trick question that would trip me up during a job interview:

An algorithm is said to be constant time (also written as O(1) time) if the value of T(n) is bounded by a value that does not depend on the size of the input. For example, accessing any single element in an array takes constant time as only one operation has to be performed to locate it. However, finding the minimal ...

Read More Source

September 5th, 2016

In Technology

No Comments

Beware Big O Notation in higher level languages

This essay has a nice bit of details about a general point that needs to be made:

Here we get the result that will be counterintuitive to many. No matter how large n gets, the Array List still performs better overall. In order for performance to get worse, the ratio of inserts to iterations has to change, not just the length of the collection. Note that isn’t an actual failure of Big O analysis, it is merely a common human failure ...

Read More Source

September 5th, 2016

In Technology

No Comments

The gamification of Slack makes it addictive but also makes it a drain

Very interesting:

Everything bad about Slack is fundamentally Slack’s fault. Slack’s sane default for a new user is to play a sound and send a desktop notification every time anything happens anywhere. There is no way to simply turn off the screaming red circle on the dock icon. There is no way to simply fold away the sidebar so you can focus on whatever it is you are trying to do. These UI nitpicks help us understand Slack’s conception of the ...

Read More Source

September 5th, 2016

In Technology

No Comments

It’s not even XML!

Dave Winer said he is deeply depressed. Sometimes he seems frightened by the march of change. I suppose that is an aspect of depression. His initial fear of JSON seems like an expression of fear:

I’ve been hearing, off in the distance, about something called JSON, that proposes to solve a problem that was neatly solved by XML-RPC in 1998, the encoding of arrays and structs in a format that could easily be processed by all programming languages. The advantage ...

Read More Source

September 5th, 2016

In Technology

No Comments

Finding novelty via friends is a metric not yet captured by recommendation engines

This is a good point:

The traditional regime of recommendation systems has been obsessed with (1). What’s the uplift of recommendation algorithm A vs recommendation algorithm B? Which is driving more click-thrus and conversions?

There’s something fundamentally broken in the this way of thinking though. I don’t care what the computer says. I care about what my friends say. The meaningful music I’ve discovered over the last 10 years has been music liked by someone I respect. Friends. Other musicians I ...

Read More Source

September 5th, 2016

In Technology

No Comments

Detecting voting rings with HyperLogLog

While this seems like a clever trick, I typically want a lot of metrics regarding voting, so throwing away the metadata doesn’t seem like an option to me. As an aggregate tool whose only purpose is finding voting rings, maybe this useful maybe?

But consider what would happen if we created a HyperLogLog counter for every user on Reddit, and any time that a user receives an upvote, we update the corresponding HyperLogLog counter with the id of the user ...

Read More Source

September 5th, 2016

In Technology

No Comments

Great browser software in 1996

These technologies sound so amazing, I wonder what happened to them?

Atlas is the precursor to the next step in Navigator’s evolution, Version 3.0. It’s “alpha” code; in other words, that celebrated tower in Pisa is more stable. When it works, Atlas promises to deliver:

VRML viewing. VRML (Virtual Reality Modeling Language) is one of the competing standards for expressing three-dimensional information in compact form. VRML documents can be static or interactive. For example, you could create a Web page that ...

Read More Source

September 5th, 2016

In Technology

No Comments

The Big O cheat sheet

I just discovered the Big O cheat sheet and I think this is very useful:

Post external references 1http://bigocheatsheet.com/ Source

September 4th, 2016

In Technology

No Comments

Faraday’s breakthrough came when he wrapped two insulated coils of wire around an iron ring

I just realized that I’ve been confusing Faraday and Maxwell. It’s Maxwell who did the 4 equations.

Faraday’s breakthrough came when he wrapped two insulated coils of wire around an iron ring, and found that upon passing a current through one coil a momentary current was induced in the other coil.[2] This phenomenon is now known as mutual induction.[44] The iron ring-coil apparatus is still on display at the Royal Institution. In subsequent experiments, he found that if ...

Read More Source

August 19th, 2016

In Technology

No Comments

HAProxy for MySQL

Three years ago, I was intrigued to read that Twitter had implemented dynamic network topologies. Something similar is becoming more common, thanks to the use of Nginx/HAProxy. One can use this for many things, such as connecting to a pool of MySQL servers.

However, there are some downsides worth remembering:

You have to be careful to tune your xinetd script to increase the cps and per_source limits accordingly to the load that your pool of servers generate with this proxy forwarding. Assuming that ...

Read More Source

August 19th, 2016

In Technology

No Comments

Clever uses of a custom TCP stack

Interesting:

Fun anecdote, at Blekko we had people who tried to scrape the search engine by fetching all 300 pages of results. They would do that with some script or code and it would be clear they weren’t human because they would ask for each page right after the other. We sent them to a process that Greg wrote on a machine that did most of the TCP handshake and then went away. As a result the scrapers script would hang ...

Read More Source

August 16th, 2016

In Technology

No Comments

When algorithms betray you

Interesting:

You decide to share an article about Brexit from “The Atlantic”, which will somehow shed light to all your friends as to why it happened. The article is 1,000 words long — you only read half of it, but that’s good enough. It captures all the arguments you’ve been wanting to make for the past two months to your friends. Will this be the Facebook post that finally spurns your friends into action? You realize your Facebook friends all agree with your ...

Read More Source

August 11th, 2016

In Technology

No Comments

Only good leadership can create good software

This is very true, and it applies to more than software:

Thirteen years ago, Eric Raymond’s book The Cathedral and the Bazaar (O’Reilly Media, 2001) redefined our vocabulary and all but promised an end to the waterfall model and big software companies, thanks to the new grass-roots open source software development movement. I found the book thought provoking, but it did not convince me. On the other hand, being deeply involved in open source, I couldn’t help but think that it ...

Read More Source

August 11th, 2016

In Technology

No Comments

Clever hacks whereby sites can see aspects of your browser history

Very interesting hacks:

Up until mid-2010, any rogue website could get a good sense of your browsing habits by specifying a distinctive :visited CSS pseudo-class for any links on the page, rendering thousands of interesting URLs off-screen, and then calling the getComputedStyle API to figure out which pages appear in your browser’s history.

After some deliberation, browser vendors have closed this loophole by disallowing almost all attributes in :visited selectors, spare for the fairly indispensable ability to alter foreground and background ...

Read More Source

August 8th, 2016

In Technology

No Comments

The control mania of Scrum

This is very good:

Scrum inhibits deep understanding and innovation This is actually my biggest gripe about Scrum. As mentioned above, in Scrum, the gods of story points per sprint reign supreme. For anything that doesn’t bring in points, you need to get the permission of the product owner or scrum master or someone who has a say over them. Refactoring, reading code, researching a topic in detail are all seen as “not working on actual story points, which is what you ...

Read More Source

August 8th, 2016

In Technology

No Comments

The daily Scrum meeting is a pointless ritual

I love this:

The daily standup deserves a blog post of its own. This religious ritual has become a staple of every team in the world. Ten minutes of staring into the void, talking about what you did while no one else listens, because they were in the middle of something five minutes ago and will go back to it in another five minutes, and waiting for everyone else to finish. I know this sounds cynical, but it is the ...

Read More Source

August 8th, 2016

In Technology

No Comments

If you measure programmers by points then they will optimize for points

The worst thing with Agile/Scrum is that you can hit all your 2 week targets and yet a year later have software that no one wants to use, so to the extent that Agile is suppose to reduce risk, it fails completely.

But why does it fail? This is good:

No matter how you define story points, the real issue with them doesn’t go away. The main purpose of points is making planning more reliable, and providing a temporal perspective ...

Read More Source

August 8th, 2016

In Technology

No Comments

Problems with the Scrum process

This is a great conversation:

cageface says:

But the author’s criticisms of the incentives of Scrum are on point I think. Because the stories are always articulated in terms of user facing features they encourage developers to hack things together in the most expedient way possible and completely fail to capture the need to address cross cutting concerns, serious consideration of architecture, and refactoring.

This is how you can get two years into a project and have managers and clients that think that ...

Read More Source

August 3rd, 2016

In Technology

No Comments

The JustWorks job interview

One of the most popular posts I’ve written is “Embarrassing code I wrote under stress at a job interview“. People get a laugh out of the dumb things I do doing job interviews. Here is another such post.

I did a job interview at JustWorks. They asked me to write the code to solve this problem:

The cost of a stock on each day is given in an array, find the max profit that you can make by buying once ...

Read More Source

August 3rd, 2016

In Technology

No Comments

Is Docker ready for production?

I prefer uberjars to Docker, and I prefer fat binaries, such as those allowed by Go, over Docker. And still there is the question “Is Docker ready?”

Senex says:

I’ve been tracking the beta for a while. I’m confused about this announcement. These issues still seem unresolved? (1) docker can peg the CPU until it’s restarted

(2) pinata was removed, so it can’t be configured from CLI scripts

(3) it’s not possible to establish an ip-level route from the host to a container, which ...

Read More Source

July 27th, 2016

In Technology

No Comments

How to package dependencies in Golang

Very interesting, as I’m a fan of the idea of “vendoring”:

An attempt to fix packaging in Go:

Manul is a vendoring utility for Go programs.

What’s the reason for yet another utility?

Because all other vendor utilities suffer from the following:

Some wrap the go binary and spoof the GOPATH env variable. You will have a non-go-gettable project which needs additional software in order to compile and run;

Some copy the source code of dependencies into the vendor directory:

It will be nearly impossible to find ...

Read More Source

July 26th, 2016

In Technology

No Comments

Google Maps does not name Nachitschewan?

I can not find a zoom level where Google Maps names Nachitschewan. Is this a political thing?

Source

July 25th, 2016

In Technology

No Comments

The advantage of a weight-balanced B-tree

A very interesting data structure of which I know nothing:

Post external references 1https://www.quora.com/Does-a-data-structure-exist-which-supports-logarithmic-insertion-removal-and-constant-time-query-for-the-Kth-smallest-element Source

July 22nd, 2016

In Technology

No Comments

A string of characters is best parsed using a finite state machine

I already knew this, but I don’t think I knew the extent of what was sacrificed to give Perl backreferences s:

Notice that Perl requires over sixty seconds to match a 29-character string. The other approach, labeled Thompson NFA for reasons that will be explained later, requires twenty microseconds to match the string. That’s not a typo. The Perl graph plots time in seconds, while the Thompson NFA graph plots time in microseconds: the Thompson NFA implementation is a million ...

Read More Source

July 21st, 2016

In Technology

No Comments

The problems with Clojure

The conflict between Object Oriented Programming and Functional programming comes up a lot with Clojure, because to do any serious work you have to eventually use some Java, which takes you back into the world of Object Oriented Programming. This sums up my feelings:

The thing I like most about Elixir is the low friction between it and it’s host language, Erlang. Erlang is a functional language right from the start, and the BEAM is designed to run a functional ...

Read More Source

July 21st, 2016

In Technology

No Comments

The RegX that killed Stack Overflow

A great post-mortem of a crash at Stack Overflow:

The regular expression was: ^[\s\u200c]+|[\s\u200c]+$ Which is intended to trim unicode space from start and end of a line. A simplified version of the Regex that exposes the same issue would be \s+$ which to a human looks easy (“all the spaces at the end of the string”), but which means quite some work for a simple backtracking Regex engine. The malformed post contained roughly 20,000 consecutive characters of whitespace on a ...

Read More Source

July 21st, 2016

In Technology

No Comments

The difference between Kafka and Kestrel

For my purposes, for the next 6 months, it seems that Kestrel will be all I need. I’d have to be very successful before I would need anything as complex as Kafka.

The biggest conceptual difference is that Kestrel is a simple stand alone queue where as Kafka is a full fledged queuing system.

Kestrel runs on a single machine and has no concept of clustering or failover or any other features you might expect in a queuing system. Instead the clients ...

Read More Source

July 21st, 2016

In Technology

No Comments

RabbitMQ is difficult, Kafka is awesome, Kestrel is beautifully simple

A great review of these 3 queues:

Not mentioned below is that RabbitMQ works hard to guarantee delivery of a message, so it is slow, but that is because it is in some ways doing more than Kafka.

RabbitMQ:

I created 4 queues, wrote a ruby client and started inserting messages. I got a publishing rate of about 20k/s using multiple threads but I got a few stalls caused by the vm_memory_high_watermark, from my understanding during those stalls it writing to disk. ...

Read More Source

July 21st, 2016

In Technology

No Comments

Kestrel is as simple as Redis but Kestrel is an actual queue

I feel stupid that I didn’t look at Kestrel sooner. I feel especially stupid that I’ve been using Redis when I didn’t actually need it.

Redis has a beautifully flexible API which makes it tempting to use Redis for everything, but Redis is really a cache that focuses on speed above all else. Everything in Redis has to fit in memory, and Redis will drop anything that can’t fit in memory. Also, the stuff in Redis can be mutated, ...

Read More Source

July 21st, 2016

In Technology

No Comments

Only use Amazon SQS if you need a high-latency high-concurrency service

Also interesting, don’t use Amazon SQS unless your needs fit this very specific model where you can deal with the latency and make up for it by being highly concurrent.

I just did some benchmarking the other day to compare Amazon SQS with RabbitMQ. Publishing and consuming 10,000 messages serially in 2 threads (one publishing the other consuming) on an EC2 instance took over 6 minutes using SQS and 12 seconds using RabbitMQ. I didn’t test ActiveMQ since their clustering is ...

Read More Source

July 21st, 2016

In Technology

No Comments

For Apache Storm, use Kestrel

There is a lot in this thread that is interesting. This is coming from Nathan Marz, the guy who invented Storm. He says he uses Kestrel, so that is a big endorsement.

You want to make sure that your spout source can support the out of order acking that Storm requires for guaranteed message processing. We use Kestrel because it has this property and is the simplest. RabbitMQ is another good one to consider.

Post external references 1https://groups.google.com/forum/#!topic/storm-user/rd3JD1p0wns Source

July 20th, 2016

In Technology

No Comments

The problem with RabbitMQ

Seems like RabbitMQ is good but not perfect:

Much like you’d chuck memcached on each of your web servers and access them in a ring, Darner can occupy a small niche on each box in your fleet’s resources. Tens of MB of RAM and negligible CPU opens up hundreds of gigabytes of queue spool per node. As queue size grows, memory usage remains constant.

Contrast this with Redis, which is speedy but limited in queue size to what will fit ...

Read More Source

July 20th, 2016

In Technology

No Comments

ZeroMQ is the opposite of a queue

Reading this again, several years after the first time I read it, and now it occurs to me: I would use ZeroMQ only in exactly those situations where I would not use a queue. If I only need a queue, I can use Kestrel. But if I need some messaging pattern that is not supported by any queue, then I would use ZeroMQ. That is, if I had a truly unique situation that needed a unique pattern, then ZeroMQ would ...

Read More Source

July 20th, 2016

In Technology

No Comments

Everything you need to know about queues

This looks like an amazing resource and I’m sure I’ll go back and read more soon.

Post external references 1http://queues.io/ Source

July 20th, 2016

In Technology

No Comments

The New York Times uses RabbitMQ

RabbitMQ is written in Erlang, so I was inclined to think well of it, though I heard criticism of it. And then the New York Times used it. A very surprising vote of confidence in RabbitMQ:

This architecture – Fabrik – has dozens of RabbitMQ instances spread across 6 AWS zones in Oregon and Dublin. The instances are organized into “wholesale” and “retail” layers. Connection to clients is via websockets/sockjs.

Upon launch today, the system autoscaled to ~500,000 users. Connection times ...

Read More Source

July 20th, 2016

In Technology

No Comments

Gauges uses Kestrel which is based on work from Blaine Cook

Why does Kafka get all the attention if Kestrel is so reliable? I assume this is because Kafka can do so much more, though of course the devops work of Kafka can be frightening. I didn’t know about the Blaine Cook connection (of Twitter fame):

Before I get too far a long with this fairy tail, let’s talk about Kestrel — what is it and why did I pick it?

Kestrel is a simple, distributed message queue, based on Blaine Cook’s ...

Read More Source

July 20th, 2016

In Technology

No Comments

Kestrel versus Resque

Interesting note about these queues, which certainly convinces me to use Kestrel:

There are a lot of things that could be done with either Kestrel or Resque. Because Resque is backed by Redis, you have to remember that all of the messages waiting to be processed have to be able to fit in the RAM of the Redis server, with Kestrel you could queue millions or billions of messages and then start to pull them off.

The biggest difference between the ...

Read More Source

July 20th, 2016

In Technology

No Comments

How bad can MongoDB be with Dirty Reads?

Very worrisome, though no sane person would use MongoDB to track something involving money. That’s simply not what MongoDB is for.

How bad are dirty reads?

Read uncommitted allows all kinds of terrible anomalies we probably don’t want as MongoDB users.

For instance, suppose we have a user registration service keyed by a unique username. Now imagine a partition occurs, and two users–Alice and Bob–try to claim the same username–one on each side of the partition. Alice’s request is routed to the ...

Read More Source

July 20th, 2016

In Technology

No Comments

How to return a disallowed field in GraphQL

Best Practices for GraphQL are still in a very immature stage, but these two ideas both have some merit:

First, return null for the requested field. This seems to work well in cases where there is no real harm in asking for a particular set of data and no real harm in denying it.

A good example would be asking for the email of a user where the backend only provides the user’s email to that user themselves. If I request my ...

Read More Source

July 20th, 2016

In Technology

No Comments

Why has Google released gRPC

This is interesting:

Google has been using a single general-purpose RPC infrastructure called Stubby to connect the large number of microservices running within and across our data centers for over a decade. Our internal systems have long embraced the microservice architecture gaining popularity today. Having a uniform, cross-platform RPC infrastructure has allowed for the rollout of fleet-wide improvements in efficiency, security, reliability and behavioral analysis critical to supporting the incredible growth seen in that period.

Stubby has many great features – ...

Read More Source

July 13th, 2016

In Technology

No Comments

Why I hate Ruby On Rails

I’m trying to help a friend with a very small Ruby On Rails project, where she needs a few elements of the interface tweaked. But this is Ruby On Rails, so of course I’m losing 3 hours trying to get setup. I’ve been drawn into the endless swamp of upgrading various libraries, which seems to be an automatic part of dealing with Ruby On Rails. If this was Clojure, or NodeJS, setup would have taken 5 minutes and I would ...

Read More Source

July 13th, 2016

In Technology

No Comments

Mixins are harmful — the Javascript edition

Over the years there have been arguments against mixins, but the dominance of Object Oriented Programming meant that mixins still survived and had theoretical justification. Now, the spread of the Functional Paradigm has put renewed pressure on mixins. Dan Abramov at Facebook just came out with an essay against mixins:

“How do I share the code between several components?” is one of the first questions that people ask when they learn React. Our answer has always been to use ...

Read More Source

July 10th, 2016

In Technology

No Comments

Edges is just a wrapper around the real results that adds extra data used for slicing the result cursor

I think the “edges” and “nodes” terminology is one of the most confusing things about React / Relay / GraphQL, at least at first:

Relay connection defines a relationship an an object that has two root fields: edges and pageInfo. Edges is just a wrapper around the real results that adds extra data used for slicing the results (cursors), PageInfo has metadata related to the current page. This is how it looks like:

connectionName { edges { ...
Read More Source

July 10th, 2016

In Technology

No Comments

Make the Web compatible?

It must be 1999 again because someone is writing about browser incompatibilities:

Users won’t switch browsers, they’ll switch sites

You might think that users will switch browsers to use your site. But many won’t or can’t.

Users have no patience for things that don’t work, and they’ll just go to a competitor’s site instead. Failing at a critical point could turn a potential user away forever. According to Akamai,

32% of users who encounter a problem on your site are less likely to make ...

Read More Source

June 30th, 2016

In Business, Technology

1 Comment

The Agile process of software development is often perverted by politics

For those of you who don’t want to read this whole essay, here is the summary:

The word “agile” has a plain meaning in standard English, and that meaning was considered something positive by software developers, so much so that the most successful new development process of the last 30 years calls itself “Agile”. However, at many of the companies that I have worked, actual agility is suppressed because of various political factors. Fear wins out over trust. Instead of actual ...

Read More Source

June 29th, 2016

In Technology

No Comments

How much time and energy should a computer programmer spend learning a proprietary server system?

By itself, this is an anecodote about Google’s particular system: (but see my point at bottom)

Is there a lesson in this? Well, if I were in the business of programming Google App Engine, a few days’ effort up front to get it going might seem not to be a big deal. However, when I visit a team who have just added a new person, I invariably find that person struggling to set up their workstation. There’s usually someone in the room ...

Read More Source

June 29th, 2016

In Technology

No Comments

Is Phoenix going to take over the tech industry?

If not, why? This has been one of the big puzzles of the last 30 years: if some technologies are clearly superior, then why don’t they take over? Apparently because they lack an element that appeals to management in big corporations? That’s always been the argument against Lisp: it is great for the individual master craftsman, but it doesn’t work in a big corporation full of badly paid, mediocre programmers.

Erlang has been the most safe, resilient technology out there ...

Read More Source

June 23rd, 2016

In Technology

No Comments

Pagination with React / Relay / GraphQL

This is an awesome article:

Relay proposes a standard to define a has-many relationship for a GraphQL field. This standard defines a common structure that allows Relay to paginate and filter the results in an efficient way by using cursors, which I’ll explain in a bit.

This is the definition of a Relay connection (from the Relay connection specs):

Relay’s support for pagination relies on the GraphQL server exposing connections in a standardized way. In the query, the connection model provides a standard ...

Read More Source

June 15th, 2016

In Technology

No Comments

Demanding ssh key login to a server is safer than allowing passwords

This is good and true:

Require ssh key authentication

We tend to avoid passwords for logging into servers. There was a lot of discussion around this after Bryan’s original guide came out, but I tend to fall into this camp as well. Here are a few notes on this:

ssh keys are better than passwords only because they contain and require more information.

Passwords can be brute forced. Guessing a public key is so essentially impossible that they can be considered perfectly secure

What ...

Read More Source

June 14th, 2016

In Technology

No Comments

Object/Relational Mapping is rarely worth it

I think this can be stated simply: for your basic operations (CREATE, UPDATE, READ, DELETE) then an ORM offers some convenience. But ORMs break down as soon as you have a complex query. And yet, if you start using an ORM because it helps you with the simple queries, it will seem natural to you to extend it and try to use it for the complex queries.

This is a strained analogy to the Vietnam war:

Object systems are typically characterized ...

Read More Source

June 11th, 2016

In Technology

4 Comments

AWS is inappropriate for small startups because its complexity demands a specialist

Sean Hull references a conversation that he and I had in Slack. I would like to expand on the argument that I made then. You might want to read his essay first, where he makes these points about AWS:

1. Over 70 services offered

2. Still complex to build high availability

3. Need a dedicated devops

4. Orchestration involves many moving parts

5. Troubleshooting failed deployments is difficult

At the time of our conversation I had crashed an AWS instance and we were having trouble fixing ...

Read More Source

June 11th, 2016

In Technology

No Comments

GraphQL is the best thing about React / Relay / GraphQL

React / Relay / GraphQL:

The big breakthrough here is obviously GraphQL. Not React or Relay, since there are many competing implementations that do similar things, but GraphQL. Even though there are graph query languages out there (Gremlin, etc) they were not suited to querying JSON over the wire. GraphQL is ideal. For my next project, I hope to do a Clojure implementation of most of GraphQL, because I think it can be married to Om.Next in a very powerful way.

Actually, ...

Read More Source

June 10th, 2016

In Technology

No Comments

Krubner, you saved my day

Aw man, after all the mistakes I make, it is nice to know I help people sometimes:

Post external references 1https://github.com/ptaoussanis/timbre/issues/124 Source

June 10th, 2016

In Technology

No Comments

Sometimes I make a bad decision

It’s very, very, very rare, almost unheard of, but sometimes I make a mistake.

I was helping at Open Road this spring. Sean Hull was also helping. Open Road runs a bunch of WordPress sites, such as The Line Up. All of the images for these sites were stored on a central NFS server. All of the machines were starving for hard disk space. The previous Director Of Technology had made many, many bad decisions, including the decision to save ...

Read More Source

June 7th, 2016

In Technology

No Comments

The Belief that Tests are Smarter than Code Telegraphs Latent Fear or a Bad Process

This part is really good:

Programmers have a tacit belief that they can think more clearly (or guess better) when writing tests when writing code, or that somehow there is more information in a test than in code. That is just formal nonsense. The psychological perspective is instructive here, and it’s important because that — rather than any computational property — most drives developer behaviour. If your coders have more lines of unit tests than of code, it probably means one of several things. They may be ...

Read More Source

June 7th, 2016

In Technology

No Comments

The problem with unit tests

Interesting:

Unit testing is of course not just an issue in object-oriented programming, but the combination of object-orientation, agile software development, and a rise in tools and computing power has made it de rigueur. As a consultant I often get questions about unit testing, including this real one from a recent client of mine, Richard Jacobs at Sogeti (Sogeti Nederland B.V.): My second question is about unit tests. If I remember correctly you said that unit tests are waste. First, I was surprised by that. Today however, my ...

Read More Source

June 6th, 2016

In Technology

No Comments

Ruby On Rails is obsolete

This is a strange conversation to be having in the year 2016. Ruby On Rails changed the technology industry in 2004, but it can not claim to be winning in 2016. It still lacks a story for concurrency. It is being undermined by 2 forces:

1.) the need for concurrency, which is offered by other eco-systems, such as those that run the JVM. A language like Clojure offers the high level of meta-programming that Rubyists love, but with the vast arsenal ...

Read More Source

June 2nd, 2016

In Technology

No Comments

Sometimes peer review delays progress for many decades

This is what real leadership looks like:

Morris intentionally consumes H. pylori. Like Marshall, he becomes ill, but unlike Marshall, he is not completely cured by antibiotics. The infection will remain with him for three years.

I wish he had been given a medal for his self-sacrifice. He was attempting to save millions from suffering.

Interesting, though very disappointing:

1881 Klebs notes the presence of bacteria-like organisms in the lumen of the gastric glands.

1889 Walery Jaworski describes spiral organisms in sediment washings ...

Read More Source

May 22nd, 2016

In Technology

No Comments

Why does Rails become awful?

Interesting:

Let me tell you a story about what can happen in a convoluted Rails codebase. Once, I joined an existing project. It was a huuuuge app which was running an on-line shopping community website. Complicated sales model, complicated promotions, complicated product setups, coupons, user groups, messages – it had it all. I joined them to help ship a few new features. One of my early tasks was to…add a link to something on some page. It took me few days ...

Read More Source

May 15th, 2016

In Technology

No Comments

The dumbest name for a good architecture is “serverless”

I strongly agree that the idea is good, but the name “serverless” is really stupid:

So… this is backend infrastructure (i.e. it runs on a server) to let your team deploy “severless” apps…

So they’re serverless. Except the server you’re running it on. And the lamba/style code they wrote and uploaded to it.

But serverless, because no ops staff required. Except the ones who installed and maintain this.

This is like a snake eating its own tail and wondering what hurts.

Edit: despite my sarcasm, ...

Read More Source

May 10th, 2016

In Technology

No Comments

The sad, slow way a system of cron scripts becomes ugly

Do you have a chore that needs to run in the background, maybe once a day, or once an hour? Cron scripts will save you! They are the most amazingly amazing thing God has invented since Adam and Eve! For sheer wonderfulality they have no equal among the products of mortal fallen flesh!

At least at first.

The simple cron script is wonderfully direct and efficient. But the first can lead to a second. The second can lead to a third. ...

Read More Source

April 10th, 2016

In Technology

No Comments

What do AWS services actually do?

I love this:

Data Pipeline Should have been called Amazon ETL

Glacier Should have been called Really slow Amazon S3

Kinesis Should have been called Amazon High Throughput It’s like Kafka

I have used this but I did not know the name:

Snowball Should have been called AWS Big Old Portable Storage Use this to Get a bunch of hard drives you can attach to your network to make getting large amounts (Terabytes of Data) into and out of AWS It’s like Shipping a Network Attached Storage device to AWS

These are useful names that actually describe what ...

Read More Source

March 27th, 2016

In Technology

No Comments

The fanaticism of “test driven development” is slowly fading

This was one of those trends that went too far and I’m glad to see people talking about reasonable limits:

If you’re part of the Ruby On Rails community for a long time, you’ve probably read tons of articles about testing Rails application (less these days, though). Although there always have been diverging opinions on the matter, it seems the common wisdom was to say that you had to test everything: models, controllers, views and full-stack tests. Oh, and you had ...

Read More Source

March 26th, 2016

In Technology

No Comments

RESTful APIs are dead, long live GraphQL

We can argue whether RESTful APIs ever existed, since very few ever implemented HATEOAS. Ruby On Rails claimed it offered RESTful APIs, but it never offered HATEOAS. After 10 years of failure, the world is looking for something new.

Interesting:

Imagine we have a simple application that fetches a list of stories, and some details about each one. Here’s how that might look in resource-oriented REST:

// Fetch the list of story IDs but not their details: rest.get('/stories').then(stories => // This ...
Read More Source

March 26th, 2016

In Technology

No Comments

Picking React isn’t a technology decision, it’s a business decision

Interesting:

The clear winner right now, is React. Components all the way down makes your application much easier to reason about. The learning curve is very flat. The important APIs would fit on one page. JSX is awesome. You get all the power of JavaScript and its tooling when writing your markup. It is the natural match for Flux and Redux (more on that later). The React community is amazing, and produced many best of breed tools such as Redux (also more on that later). Writing high ...

Read More Source

March 26th, 2016

In Technology

No Comments

Pay discrimination in tech

This sounds bad:

Women who write the software that runs on mainframe computers earn on average 72 cents per dollar earned by their male counterparts, according to research conducted by Glassdoor Inc., the online job information firm. That pay gap exists even after controlling for age, education, experience, job title, employer and location.

This suggests that the gap widens with age:

However, not all tech jobs pay men and women so differently. Among mobile developers, there is just a 2.9% gap between ...

Read More Source

March 26th, 2016

In Technology

No Comments

Left-Pad as a service

Everyone is laughing over this. An 11 line function as its own NPM module? I am unsure how I feel. The trend has been to smaller and smaller pieces of functionality. I don’t find this outrageous. Still, the parody is funny:

c4n4rd is game:

This is really exciting!!! I was a bit disappointed that the right-pad will be out only in 2017. I am looking forward to that release because there is a high demand for it now.

What kind of ...

Read More Source

March 26th, 2016

In Technology

No Comments

null

Testing WordPress to see if a title of “Null” works.

Source



Check out my books:










RECENT COMMENTS

February 8, 2022 9:33 am

From Michael S on How I recovered from Lyme Disease: I fasted for two weeks, no food, just water

"Did you have Bartonella, too? Seems it uses autogenesis..."

January 11, 2022 4:33 am

From Essie on Docker is the dangerous gamble which we will regret

"Once in 1990s, there are popular high performance solution called HPC software, many commercial softwares are ..."

December 17, 2021 7:32 pm

From John Carston on The ethics of being a high level tech consultant (a Fractional CTO)

"It helped when you mentioned that it is important to have a real connection with your consumer. My cousin ment..."

September 2, 2021 7:47 pm

From Mojavedfo on Where PHP regex fails

"55 thousand Greek, 30 thousand Armenian..."

August 7, 2021 9:53 am

From Colin Steele on The ethics of being a high level tech consultant (a Fractional CTO)

"Fantastic essay. Thoughtful, well-constructed, timely and applicable. I think every part-timer in the tech f..."

August 5, 2021 3:02 pm

From Rachiovwn on Where PHP regex fails

"consists of the book itself..."

October 19, 2019 3:08 am

From Bernd Schatz on Object Oriented Programming is an expensive disaster which must end

"I really enjoyed your article. But i can't understand the example with the interface. The example is reall..."

October 17, 2019 4:50 pm

From Anderson Nascimento Nunes on The conventional wisdom among social media companies is that you can’t put too much of the onus on users to personalize their own feeds

"Can't speak for anyone else, but on my feed reader: 5K bookmarked feeds, 50K regex on the killfile to filter o..."

October 10, 2019 11:17 am

From روابط: البث المباشر – صفحات صغيرة on RSS has been damaged by in-fighting among those who advocate for it

"[...] تاريخ تقنية RSS، مقال قديم ويلقي نظرة على الناس الذين طوروا التقنية [...]..."

October 9, 2019 3:08 pm

From Dan Campbell on Object Oriented Programming is an expensive disaster which must end

"Object-Oriented Programming is Bad https://www.youtube.com/watch?v=QM1iUe6IofM..."

October 4, 2019 8:44 pm

From lawrence on My final post regarding the flaws of Docker / Kubernetes and their eco-system

"Gorgi Kosev, I am working to clean up some of my Packer/Terraform code so I can release it on Github, and then..."

October 4, 2019 5:14 pm

From Gorgi Kosev on My final post regarding the flaws of Docker / Kubernetes and their eco-system

"> Packer, sometimes with some Ansible. The combination of Packer and Terraform typically gives me what I ne..."

October 4, 2019 12:40 pm

From lawrence on My final post regarding the flaws of Docker / Kubernetes and their eco-system

"Gorgi Kosev, about this: "I would love if you could point out which VM based system makes it simpler and..."

October 4, 2019 7:31 am

From Gorgi Kosev on My final post regarding the flaws of Docker / Kubernetes and their eco-system

"I won't list anything concrete that you missed, because that will just give you ammunition to build the next a..."

October 4, 2019 1:39 am

From lawrence on My final post regarding the flaws of Docker / Kubernetes and their eco-system

"Gorgi Kosev, also, I don't think you understand what a "straw man argument" is. This is a definition from Wiki..."