Author Archive

Sean Ryan

Developer for Troy Web Consulting.

Language Unification for In-House Application Development: Part II

Written by Sean Ryan on . Posted in App Dev

A Guide to selecting the most appropriate web language for your organization

In Part I of this series, we looked at the importance of investing some time into selecting a programing language for your organization and the types of questions that should be discussed with your group throughout the process. In Part II, we paint a fairly typical scenario of an organization in which the decision to rally behind one programming language has been made but the question of which language remains unanswered.

Part II

We start by listing the more popular Web programming languages today and work from there. The list is presented in alphabetical order and in no way represents a preference for one over another.

Note

The following table and brief explanations of “popular languages today” is a subset of the possible languages to use when writing web apps and to an extent, desktop apps. The list was carefully selected and excludes such languages as C/C++, Perl, Erlang, and Clojure/Lisp. This article assumes that if your organization is in consideration of language other than the nine languages that follow, its needs are outside of the scope of this article.

 

Language Use Programming Style Runs on Year/Age Frameworks
ASP.NET Web, server side scripting Object oriented CLR 2002/12 ASP.NET MVC, Spring.NET
ColdFusion Web, server side scripting Imperative/Procedural with support for Object oriented JVM, CLR 1995/19 CF Wheels, ColdBox, FuseBox, Mach-II, ModelGlue, some others
Groovy General purpose, client and server side scripting Object Oriented, functional, and imperative JVM 2007/7 Grails
Java General use; Desktop, Web, Client/Server Object oriented and imperative JVM 1995/19 Spring (specifically MVC), JSF, Struts, GWT, Tapestry, many more
Node.js (JavaScript) Client/Server, Web, Real-time applications, scripting Object oriented, imperative, and functional Node.js / Google V8 Engine 2009/5 Express, Geddy, some others
PHP Web, server side scripting Object oriented, functional, procedural, and imperative PHP Interpreter 1995/19 Laravel, Phalcon, Symphony2, CodeIgnitor, many more
Python General purpose, scripting, Web , Desktop Object oriented, functional, procedural, and imperative Various compilers, VMs, and interpreters 1991/23 DJango, Grok, some others
Ruby General purpose, Web Object oriented, functional, and imperative Various Ruby interpreters 1995/19 Rails
Scala General purpose Functional, object oriented JVM 2003/11 Lift, Slinky, Sweet, some others

 

Now that we have narrowed the field of options from seemingly hundreds to nine likely candidates, let’s review a realistic scenario for which an analytical selection can be made.

Scenario

You are the division Director of Information Technology for a medium sized government agency. This scenario applies equally as well for non-profits. The agency’s core mission is not building software but the division is an integral, and therefore critical part of the providing daily services required by the agency.  The software developed in-house is roughly 80% internal facing and 20% external facing. The internal community consists of 1,500 employees who use the custom software as part of their daily job duties whereas the external community comprises the entire world, yet more realistically about 20 million potential users. The developer pool is made up of about thirty programmers spread over six units/teams. The majority of programmers are familiar with Java, a small minority knows or understands the functional paradigm of programming, and everyone understands procedural/imperative as well as object oriented programming. Most internal applications are written in ColdFusion and all external applications have been developed in Java. Developer ages range from 18 to 55 but more than half are over 40.

You’re tasked with recommending to the Chief Technology Officer, a software development strategy that will 1) ease the recruitment of talented programmers by bringing the agency up to date with modern software development methods and by appealing to the up and coming programmer workforce, 2), reduce the agency’s technology fragmentation across each team and make assigning resources to inter-departmental projects easier 3) work within the existing network and server architecture with little or no modification, 4) take advantage of the existing agency talent to the largest extent possible, and 5) continue to meet or exceed the wide range of internal and external needs, from desktop applications to web portals, without impacting project deadlines and budgets.

Recommended Language and Framework

Groovy on Grails is our ultimate choice for the organization. Here’s how one might come to this conclusion.

Java

Java appears to be a natural fit. Given that there are already a number of applications running Java and ColdFusion, we know that some of the server infrastructure is already set up to deploy applications on a JVM. After all, ColdFusion is written in Java and runs on Tomcat.  We also know that most programmers in the organization know or are familiar with Java so it would be a lighter lift if we choose Java outright. Not having to train both programmer and system administrators also helps tip the scales toward Java. Following these arguments, it’s easy to lean toward one of the four languages that run natively on the JVM: ColdFusion, Java, Groovy, or Scala.

Python, PHP, and ColdFusion

We can objectively rule out Python and PHP based on the aforementioned arguments but both are perfectly acceptable language options when examined without the considerations of the scenario. Python in particular has garnered the attention and respect of web developers in recent years due to its robustness as a language.  Not only does it work well with frameworks for web applications, it can also be run as a scripting language on a variety of platforms, extending its usefulness as a tool for programmers. PHP and ColdFusion are effectively limited to the web world. Factoring in the need for web and non-web application development, languages that can be used on and off the web exhibit a more desirable quality.

.NET

ASP.NET can be ruled out based on the organization’s existing programming knowledgebase and on the existing support infrastructure. The .NET family of languages requires a Windows environment and while end users’ computers are mostly Windows, the servers are not. Introducing a new Windows infrastructure along with a new software infrastructure introduces additional technical and human resources costs. A switch to Windows-based development can be determined, for this scenario, time and cost prohibitive.

Node.js

Node.js isn’t as easily ruled out but when lined up with the remaining choices, one can start to chip away at it as the best option for the organization. For one, it’s very new and the available libraries are not as plentiful as the long established players like Java.  For another, Node apps are developed in JavaScript and JavaScript applications require a very particular style of programming. The applications process data asynchronously and rely heavily on closures, callback functions, and use a prototyping method of objection-oriented design. Those concepts can be a lot for some programmers to wrap their heads around and unless there is a compelling reason to overhaul the organization’s approach to designing software, it’s probably best to appeal to the widest skills market possible.

A Closer Look at Ruby, Scala, Java, and Groovy

Ruby, Scala, Java, and Groovy are all great contenders for the needs of this organization.

A Focus on Spring

If we select Java, the most robust framework is Spring. Spring is the most prolific Java framework series because it offers developers more than just a framework for web. There are eighteen projects that comprise Spring. They’re maintained by Pivotal Software, the same company that maintains Groovy and Grails. The framework has been so successful that it’s currently being ported to Scala as a project aptly named, Spring Scala. It operates as an IoC container, i.e., a dependency injection framework and follows the Front Controller pattern for which every resource request is first handed to Spring, pre-processed and subsequently handed off to a part of the application configured to handle that request. The exact same pattern is implemented in Groovy on Grails and in Ruby on Rails.

Due in no small part to the offerings of the Spring framework, we can rule out Ruby on Rails and Scala. Rails was an incredibly successful pioneer in the convention over configuration approach to development. Fortunately, the successes of Ruby on Rails lead to the creation of Groovy on Grails (and later, Scala) which follows the same principle but is Java-based. Most native Java code is also syntactically correct Groovy code, making the transition for a Java programmer very straight forward and much faster than learning a wildly new syntax. The Grails framework is in fact the Spring framework applied as a convention over configuration. Everything Spring offers is directly available in a Groovy on Grails application with no special integration necessary.

The out-of-the-box availability of both Java and Spring within a Groovy on Grails app, makes it an excellent fit given our scenario above. Spring was designed as a fast and flexible way to develop Java web apps. Groovy was inspired by Ruby and to address the short comings of Java. And finally, Grails is to Groovy what Rails is to Ruby. The end result is the best of all worlds while minimizing the disruptions caused by transitioning to a unified language platform across the organization.

Analysis

The scenario might read as an oddly specific set of circumstances. However, its goal is to represent the more general scenario of recognizing a need to have a consistent software development strategy; a strategy that harnesses an organization’s existing resources yet doesn’t alienate the future workforce it will need to recruit and come to rely on.

General-use vs. Nuanced Language Constructs

The fictitious organization above is far better served by selecting a general purpose language without getting into the nuances of each language’s constructs.

Each of the nine languages and their corresponding frameworks provide value to building out software. Some are faster for development (e.g., ColdFusion), some are more scalable (e.g., Scala), and still others offer very fast execution speeds (e.g., Node.js).

General-use languages like Java and Python, by their very nature, aren’t necessarily designed to solve a single problem exceptionally well. They appeal to a wider audience and allow for a variety of approaches to software design. General-use languages are excellent choices for organizations that need this sort of flexibility.

On the other hand Twitter for all intents and purposes has a single product – Twitter. Twitter’s font-end was once written in Ruby on Rails and their tweet storage written in Scala. In time Twitter moved away from Ruby on Rails in favor of Java and moved away from MySQL in favor of Lucene only to notice a 3x reduction in latency. Unlike the organization in the scenario, companies with a near singular focus – like Twitter – have the luxury of selecting a language based on very specific and pre-defined requirements.

Patterns and Frameworks

The organization in our scenario should focus on languages with several very popular frameworks that implement today’s proven patterns. Look for frameworks that provide structure to an application and encourage sensible design strategies without getting in the way of the developer. Convention over configuration is one way today’s frameworks offer a competitive edge and nearly all language-communities today have such an option.

It’s often helpful to understand the motivations for a language’s creation when deciding if it’s a language worth considering. Languages appeal to programmers for a variety of reasons. Possibly the largest factor is the speed and simplicity in which a program can be written.

Patterns of development help keep programmers focused on the business problem by laying out a proven method of attacking problems. Such patterns manifest in the form of frameworks. The Rails framework for Ruby was a pioneer in modern approaches to web development. The success of Ruby on Rails has been recognized and repeated in other frameworks. The Rails framework uses the Active Record pattern for example. This pattern greatly simplifies the pathway from the physical data model to the logical data model by coupling them directly through domain objects. This flies in the face of traditional patterns like the Transfer Object Pattern that stress a decoupling of models. Both patterns accomplish the same goal but with different layers of abstraction. Even the abstraction community giant, Java, has adopted this realization in various libraries like JOOQ and QueryDSL. Abstraction was once seen as beautiful, scalable, neatly organized code. However, abstraction and today’s need for rapid application development (RAD) have a seemingly perfect indirect relationship; not ideal. As the RAD approach to software development continues to grow in popularity, languages and frameworks need to adapt and keep up or be left behind in favor the new kids on the block. In short, change in software development is unavoidable; accept it and use it to your advantage.

Successful frameworks within and across languages copy the best parts of each other. Java inspired Scala, Ruby on Rails inspired the creation of Groovy on Grails. Rails also inspired the CF Wheels framework. When patterns work well, they’re incorporated into software design and when software is designed with standard patterns, there is a direct correlation with maintainability and in-house portability.

Language Ownership

Since the organization will have to build a large breadth of application types, it’s likely to require a wide range of libraries. It is highly recommended to select a language with a large and organized community driven code repository.

When the community is empowered to have a direct influence on a technology, be it a language or a framework, the technology has a much higher likelihood of long-term sustainability. Open source solutions have this inherently built right in but a project doesn’t necessarily have to be open source to be successful. Java for instance, while now open source, continues to rely on a Java Community Process for bettering itself. It brings together many perspectives and through diversity, creates a product developers want to use. The Apache Software Foundation is similar in this respect and its projects are all open source.

To once again use ColdFusion as an example, it’s not open source and community requests are sometimes implemented poorly or not fast enough to keep pace with competitors. This may be due in part to the road it’s taken through corporate acquisitions when Allaire was sold to Macromedia and Macromedia was sold to Adobe. At one time, ColdFusion was a singular focus of a company. Now, ColdFusion is just another product in a massive corporate portfolio (though, perhaps the same can be said of Java and Oracle).

Java has a massive library offering and it’s organized through a series of Maven repositories. Node.js, while still new on the scene, has the NPM library offerings. Most languages have something similar but not all. ColdFusion does not have any such organized library manager.

The Human Factor

It’s not always possible and practical to select a language for a project or for an entire organization based purely on market and textbook research. Doing so misses something very important: the human element. In part I of this article, I placed a notable emphasis on the programmers. Their existing skills, aspirations, and opinions matter just as much as any analytic findings. Programming is an equal mix of art and science. If a programmer is unhappy with his or her tools, the product and ultimately the organization, will suffer.

Language Unification for In-House Application Development: Part I

Written by Sean Ryan on . Posted in App Dev

A Guide to selecting the most appropriate web language for your organization

Part I

What language should our company use to develop in-house applications? A question that appears to carry with it a simple answer but too often, the decision isn’t well thought out or doesn’t originate from what makes sense for the organization as a whole.  It’s important to approach the decision from multiple points of view and to factor in more than just technology; team perspective is just as relevant.

This two part series was developed with Information Technology Management in mind with the hopes of stimulating a different way of thinking about language choice in an IT shop. Part I focuses of the process of selection. Part II breaks down a real-world scenario where an organization uses its team members’ current skill set to transition from a fragmented but mostly Java shop to a Groovy on Grails shop. While no one language is the right answer for all teams, Part II demonstrates an example of applying the principles discussed in Part I.

Introduction

The world of Information Technology is on a constant path of change and nearly all of it is out of the hands of any one organization. Organizations can opt to change with it, and continue to provide its customers with the data and information they demand or to not change and watch slowly as fewer and fewer customers turn to them for timely, relevant, accessible results.

Adopting to change is not just about maintaining a customer base. It’s also about being able to provide services that may be mandated by governing authorities, reducing the costs of processing requests, increasing the integrity input/output, providing expected or required levels of data protection, and controlling the cost to manage local IT services. A significant part of that cost is producing those services by building out in-house applications.

Once the decision has been made to develop custom in-house applications and a technology stack has been identified, the next step is to determine the most appropriate programming language for the organization or unit. Language selection should be driven by several factors and chief among them is the selection of frameworks.

Frameworks

Frameworks allow programmers to use a consistent pattern to receive, process, and produce data. They take the “busy work” out of app dev by handling events and processing parts of the request that need to be dealt with but are not directly part of the business problem.

For example, a framework should provide a way to secure a web application from unauthorized access into the system. It should allow a programmer to describe what “secure” means and what to do when a secured resource is requested. Spring security is an example of a web security framework that can be plugged into just about any Java application framework out there. Assigning a staff programmer to write the organization’s security framework is not only a waste of time; it’s also a risky idea.

A lot of smart people write and maintain frameworks and the community at large validates their usefulness by using them or not using them. Over the last twenty years or so frameworks have come and gone but several have stood the test of time and evolved into what has become a de facto requirement for building web applications.

Technological Costs

Generally, languages are free but their dependencies are not always free. For example, Adobe ColdFusion is cost $1,500 for a standard license and $8,500 for an enterprise license. However, it’s a full package solution. It ships with a pre-configured Tomcat application server and several admin tools, it runs on most major platforms, and is supported by Adobe. On the other hand Java is free. Tomcat, the most popular servlet container is free. GlassFish, the most popular application is free. Eclipse, the most popular development platform is free. Java’s runtime JRE (JVM) is free.  And finally, if running the app on some Linux distributions, the platform is also free.

Other costs include the tools necessary (or highly recommended) to write applications and to distribute application. Microsoft Visual Studio is needed to write C# and .NET applications. Distributing an iOS mobile app requires a yearly developer subscription to Apple, yet Android is completely free. One should be sure paying for a language solution is a necessary cost and in the best interest of the service that the organization will be providing. Sometimes the cost is necessary but it should be backed with quantifiable conclusions and not just antidotal observations from strong minded individuals.

Delivery of Applications

Desktop applications, web applications, network application, and mobile applications are very different and while they can all be written – to some degree – with most languages, language unification around all delivery mechanisms is a bad idea and any attempt to do so, would almost certainly manifest expensive problems shortly thereafter.

ColdFusion, for example is a relatively fast language to pick up but it’s a web application language designed exclusively for running small to medium sized web apps that use a traditional request/response paradigm. On the other hand, Java and .NET are both well suited for desktop, web, mobile, network, and just about any other use case. However, they are not as quickly learned and mastered and require external libraries/dependencies to accomplish all of the above.

Some languages are said to run on any platform. The quintessential example of this is Java, though a bit misleading. Java runs on any platform for which a JVM (Java Virtual Machine) exists. Since Java runs on a JVM and code must be further translated to the machine code it runs on, Java was viewed for many years as slow. This too is misleading since speed issues have long been resolved and are considered nearly inconsequential in today’s applications. However, for high performance network computing, C is a much better choice since it runs natively on the machine itself without any VM.  Another example is .NET. .NET applications require a Windows environment so if an organization’s infrastructure is mostly non-Windows, .NET is not a recommended choice.

It’s a good idea to choose a language that can be used for the majority of application types and one that can be run with the least amount of effort on most or all platforms.

Standards Based

The Web has been around for a little more than twenty years and web applications have been around almost as long. To say web applications have evolved in that amount of time is an understatement. Not only has the HTTP protocol changed significantly in that time, so too has the demand and expectations consumers have placed on web applications.

As more and more expectations were placed on the application, applications grew larger and more complex. Programmers trained (or not trained) in application development found themselves having to write scalable containers and complicated data pathways through their applications just to solve an otherwise simple business problem. Many were ill-equipped to write such applications and consequently some web apps lacked scalability and suffered from poor usability, buggy code, security holes.

As the software engineering community struggled to find the most effective techniques of working with this new challenge (web vs desktop) a set of standards and best practices emerged. Today, these well-tested solutions to common problems are implemented and shared with the community at large in the form of patterns, frameworks, and libraries.

Standards are critical to the success of an application, not only in producing the application, but also for the lifetime maintenance of the application. Today, developers are formally trained or otherwise learn the standard proven approaches to problem solving. Employing standards simplifies handing off a design from one developer to another and increases the predictability of the code. They also make debugging straightforward and reduce the likelihood of over engineering a solution and introducing a maintenance nightmare.

Industry and Developer Proliferation of the Language and its Frameworks

Perhaps the second biggest factor to consider when selecting a language is popularity. Honestly answering a few questions should weigh heavily on a language selection discussion.

How many [web|mobile|desktop|etc.] applications use this language?

The Java community likes to boast that over 3 billion devices run Java and they have good reason to. For better or for worse, it suggests that the language is pretty popular is likely to stick around for a while. Even if industry leaders – over night – were to tell the word Java is on its way out, the need for Java developers would not be impacted for a long time to come. After all, 3 billion devices and billions more applications would need to be maintained for at least some time. In reality, there is plenty of room for multiple languages at the forefront of industry.

How many developers in the region or your pool know this language? How long to bring a new developer up to speed?

A very important factor is availability of programmer resources. If migrating from one language to another, how many exiting team members know the new language? If hiring new developers, how prevalent are they in the community? If planning to train developers, how heavy of a lift is the new language to learn?

A programmer trained in functional programming may struggle with adjusting to an object oriented language or the MVC style of programming. Node.js is a JavaScript platform and JavaScript is a typeless language that relies heavily on closures.  Closures not easily grasped by some programmers. On the other hand Groovy is so close to Java that the learning curve for a Java programmer to learn Groovy is nearly flat and not very long.

How mature is the language? Is the adoption rate increasing or decreasing?

In the last dozen years, several popular web languages have popped up and have taken hold in certain segments of the community. Some examples include but are in no way limited to: Scala as a replacement for Java in the early 2000s that runs on the JVM, Rails (framework) in the mid 2000s as a convention over configuration web frameworks that runs on Ruby but with out the popular criticisms of Java, Grails (framework) shortly after Rails as a Groovy version with all the benefits of Rails but able to run on the JVM, Node.js in the late 2000s as JavaScript platform for building server side web apps in JavaScript.

As new languages and frameworks emerge, some languages struggle to keep up and begin to fade. For example, ColdFusion was a wildly popular for many years and to this day is still being actively used and maintained. However, its popularity is fading. It began as a procedural style language but to survive, it needed to adapt to add support for object oriented programming. By design, it never fully achieved that goal. When the RESTful style of web programming blew up in popularity as an important technique to decouple the front and back-ends, ColdFusion fell even further behind. When it finally added the RESTful feature, it was and continues to be plagued by bugs.

Programmers gravitate toward languages and frameworks that offer out-of-the-box solutions to the everyday problems they’re faced with so they don’t get in the way when solving the business problems they’re hired to solve. This cannot be understated when looking to hire talented programmers with the intent to keep them around.

Who owns the language? Is it being actively maintained? Are there add-on/plugins/libraries available? How many go-to frameworks exist for this language? How actively maintained is the framework?

This series of questions is to help answer the more general question: if the language can’t do it natively, is there a public distribution structure in place to use libraries that can do it? Many language-communities provide this feature. Java and Groovy have the Maven repositories, Node.js has the NPM repository, PERL has CPAN, Ruby has gems, Python has PyPi, and front-end JavaScript developers have Bower.

Without public support for sharing libraries and a mechanism for peer-reviewed / peer-tested code, programmers are left to reinvent the wheel time and time again. Valuable development time is wasted on developing common code. Programming is very much an art and like other artists, programmers enjoy sharing their creations with the community. Look for programming-communities where there is a groundswell of support for sharing their craft.

Committing to a Language and Framework

There are plenty of benefits to committing to a singe language or framework in an organization. Among the staff there is a consistent and predictable knowledge base. The ease of sharing or transitioning programmers from team to team is valuable for a project manager and management of a single infrastructure to support app distribution is valuable to a system administrator.

Language unification is not without its pitfalls either. It can lead to a lack of skill diversity among the organization’s programmers. It may also force a technology on a problem that may be better solved with a different language and without careful attention, it will impede of the intellectual growth of an organization’s programmers by reducing their professional exposure to other languages.

Weighing the pros and cons of selecting a common language should also take into account the purpose of the organization. A consulting firm is better served with a diverse body of languages and frameworks whereas a government agency is better served with a unified approach to app dev.

Eliminate jQuery Validation Errors in Eclipse

Written by Sean Ryan on . Posted in App Dev

Eclipse doesn’t like jQuery and each time I begin from a new install, I need to configure it to ignore validation errors on jQuery. I hate that little red x. See below.

Eclipse Validation Error for a jQuery library.

Eclipse Validation Error for a jQuery library.

 

Steps to Remove this Little Red X

  1. Select your project in the Package Explorer (or however you do it)
  2. Project…Properties
  3. In the left menu: JavaScript…Include Path
  4. In the source tab, expand the folder path that includes the jQuery problem. You’ll see that you probably are set to include all javascript and exclude none.
  5. Click Edit… on the right
  6. Click Add… in the exclusions
  7. Browse to the jQuery file. If you use a pattern, it may exclude other files so use your head and exclude only what you intend to.
  8. Click Apply.

 

No more little red x

No more little red x

SSH Keys and ssh-agent on Windows/Linux/Mac

Written by Sean Ryan on . Posted in App Dev

Why am I Writing This

Using SSH to securely connect to remote servers is well understood by anyone needing to do it. However, when you ask people to use a public/private keys to connect, it starts to get a little murkier – mainly because the process is not well understood by some folks. This article breaks down the confusion and simplifies the process to a point where you’ll wonder why you haven’t been doing this all along.

Standard SSH

Let’s assume you need to connect to server1.domain.com as user superman. You can do this with the following command.

ssh superman@server1.domain.com

This approach is fine but it involves sending your password to the server for authentication and there is no verification of who you are. Consequently, anyone with your password can get in and eavesdropping on your connection is a possibility, albeit difficult.

SSH Keys

SSH keys use the public/private key paradigm to verify the authenticity of a connection and your password is never sent to the server you’re connecting to. The private key is secret, known only to you, and stored on the client whereas the public key is well known and provided to the server. When a connection attempt is made, a challenge/response routine takes place that authenticates who you are and sets up a secured connection. This is more secure than a password and greatly reduces/eliminates the possibility of a successful brute force attack.

In fact, SSH servers can be configured to only allow communication through the use of public/private keys; AWS Linux EC2 servers default to this behavior. I’ll show how this can be done at the end of the article.

Environment

For the sake of this article, I assume you are working from a command-line interface. On Windows, I assume you are working from within cygwin. Although, the concepts are easily applied to however you are connecting.

The first thing we need to do is make sure you have the ability to create SSH keys.

On Windows

Best option: Download/install cygwin and choose OpenSSH from the package list.

Other option: download/install OpenSSH as its own application

On Mac OSX / Linux

Turn your computer on.

From a command prompt, type ssh and then hit tab twice. You should see the ssh programs we’ll be working with:

  • ssh
  • ssh-add
  • ssh-agent
  • ssh-keygen

Mac OSX

Windows

Create Your Key Pair

Creating a key pair is very simple but there is one very important guideline to remember – though not required, you should include a passphrase on your private key. Failing to use a passphrase lowers your effective security. Your private key is used to unlock encrypted messages. It’s akin to leaving the key to your house under the mat. Adding a passphrase is akin to locking your key in a lock box and putting it under the mat. I say under the mat because the typical location of these keys is pretty standard.

Weak Argument Against Using a Passphrase: I’ll have to enter a password every time I use SSH and I don’t want to 🙁

My rebuttal: Use ssh-agent and stop complaining.

To create your public/private key pair on a command line, run the following command.

ssh-keygen

The default algorithm used to create the keys is RSA. We’ll stick with defaults here. When prompted, enter a passphrase (or don’t – your choice). You should see output similar to the screen shot below.

Mac OSX ssh-keygen

You can also use a putty program on Windows called puttygen.exe. To use puttygen to generate your keys, open the program and click Generate key pair… from the Key menu.

I love the random number generation here!

You can add a passphrase and save both keys after the randomness is done.

That’s it. Now we need to share the public key with the server we’ll be connecting to. You can get the file to the server anyway you want.

HowTo: Install a Public Key on a Linux Server

Here’s a one-liner that’ll do it.

cat ~/.ssh/server1.pem.pub | ssh superman@server1.domain.com 'cat - >> ~/.ssh/authorized_keys2'

This pipes the content of your public key to SSH which in turn connects to your server. Once the connection is established, it takes the contents of standard in (result of piping content through ssh) and appends it to a file called authorized_keys2 in your account’s .ssh folder.

Note: You will be prompted for a password. This is your standard ssh password and should be the last time you’ll need to enter it in this context. Also, if .ssh directory does not exist, this command will not create one. You’ll need to do that yourself and then re-run the command.

HowTo: Install a Public Key on a Windows Server

This is a bit trickier, but only because we need to know what ssh server the Windows instance is running.

I’m Assuming WinSSHD

If you’re connecting to WinSSHD on Windows, you’ll need to import your account’s public key into the application using the graphical interface. From the control panel of WinSSHD, click “Open easy settings” from the settings area.

Control Panel of WinSSHD

For WinSSHD Virtual accounts, click the third tab – the one named Virtual Accounts. If you connect to WinSSHD using Windows accounts, click the second tab – the one named Windows Accounts.

In either case, find your username, and click the edit icon in the keys column, then click Manage. You can import your key from this screen. After you import, you’ll see your entry in the list.

Keys list in WinSSHD

Connect to the Server Using Your SSH Key

The next step is using your SSH client to connect. From a command line, run the following command.

ssh -i ~/.ssh/server1.pem superman@server1.domain.com

In putty, open SSH from the Connection drop down and then click Auth. You’ll be able to import your private key there.

Important for Putty users: if you didn’t use puttygen to generate your private key but you want to use putty for your SSH needs, you’ll need to convert your private key into a .ppk file. This is simple. Open puttygen and select Import key from the Conversions menu. Browse to your private key and and enter its password (if you supplied one). You can then save the private key in the .ppk format that putty expects. You don’t have to bother with the corresponding public key.

When you make the connection you will be prompted for a password but it is not the account password.  The password you are prompted for is for unlocking your private key. Enter it correctly and SSH will use the key to communicate with the server having never sent your actual password to the server.

Stop Repeatedly Entering Passwords

We could stop here and feel good about augmenting our connection security but we can still simplify our lives further by reducing the number of times we’ll need to enter the key’s password.

 Mac OSX

On a Mac, this really couldn’t be simpler. Not only is ssh-agent started each time your computer starts up, it also loads into itself any SSH keys it finds in your keychain. Further, when connecting over SSH to a server using a key that isn’t in the keychain, keychain will pop up and ask for the password and [optionally] store it for you. If you choose to store it, ssh-agent will never prompt you for it again. Instead, it will just get it from keychain whenever it needs it. Check it out.

By running the following command, I’m prompted to store the password in keychain.

ssh -i ~/.ssh/superman.pem superman@server1.domain.com

 

Since ssh-agent is already running, it will get the password from the keychain when I attempt to connect. This is a Mac-Specific feature and really sweet.

Now that the server’s identity has been added to ssh-agent (automattically), I never have to point to my private key again.

ssh superman@server1.domain.com

…will work without [prompting for] a password now.

Linux / Cygwin

ssh-agent still works fine with Linux and cygwin but it’s a bit more manual.

First, you need to start the ssh-agent by running the command ssh-agent.

Run the following command to add your private key identity to the ssh-agent.

ssh-add ~/.ssh/superman.pem

The screen shot below describes what’s happening.

ssh-add -D removes all identities from the ssh-agent.

ssh-add -l lists all identities in the ssh-agent.

ssh-add superman.pem add the private to the ssh-agent (notice I’m inside my .ssh directory)

ssh-add -l shows the newly added identity.

Once added, I can now ssh to server1 using the standard ssh connection string:

ssh superman@server1.domain.com

Loading ssh-agent at Startup

It’s important to note that ssh-agent is session-based. It only stores identities during the current session. As soon as your computer is shutdown/rebooted, the agent no longer contains identities.

Mac OSX: ssh-agent is started and loaded automatically. You don’t need to do anything else.

 Linux

Some distros will start ssh-agent on start up. To check if you’re currently running ssh-agent, run this command:

ps aux | grep ssh-agent

If it’s running, you’re good and don’t need to do anything else. If it’s not running, you can put the following bit of shell code in your .bashrc file.

 

Cygwin

 

Coming soon…

Adding a Polygon Search to CFMongoDB

Written by Sean Ryan on . Posted in App Dev

CFMongoDB rocks my socks…when I need to use ColdFusion.

I noticed that the CFMongoDB doesn’t natively support a polygon search 🙁 It’s ok though because I wrote one and as far as our team’s testing is concerned, it works.

I added a method to DBCollection.cfc directly under the find method called findWithin_Poly. It follows.

 

function findWithin_Poly(string loc, array poly){
  polyConverted = CreateObject("java","java.util.ArrayList").init(ArrayLen(poly));
  for ( i =1 ; i lte ArrayLen(poly) ; i++){
    polyConverted.add(poly[i]);
  }
  polyObject = variables.mongoConfig.getMongoFactory()
              .getObject("com.mongodb.BasicDBObject").init("$polygon", polyConverted);
  within = variables.mongoConfig.getMongoFactory()
              .getObject("com.mongodb.BasicDBObject").init("$within", polyObject);
  query = variables.mongoConfig.getMongoFactory()
              .getObject("com.mongodb.BasicDBObject").init(loc, within);
  var search_results = [];
  search_results = collection.find(query);
  return createObject("component", "SearchResult").init( search_results, structNew(), mongoUtil );
}

As you can see, the process is very straight forward. I build up a BasicDBObject according to the Mongo Java API and call a slightly different find method on the Mongo collection.

Parameters

The loc parameter is a string that represents the key of the document that holds the coordinates of the point. I used loc because that what I saw all over the documentation but it’s clear about not needing to be called loc.

The poly parameter is an array of arrays. Each element of the poly array is itself an array of exactly two elements: an x and a y. It represents a list of points which describe the boundaries of the polygon.

Format

The format of the poly parameter is important – as is the data in the collection. CF will automatically make your array elements strings, but they need to be doubles – java doubles.

To do this, cast them with javacast(). See the example below.

 

poly = ArrayNew(1);
polyA = ArrayNew(1);
polyA[1] = javacast("double",3);
polyA[2] = javacast("double",3);
polyB = ArrayNew(1);
polyB[1] = javacast("double",8);
polyB[2] = javacast("double",3);
polyC = ArrayNew(1);
polyC[1] = javacast("double",6);
polyC[2] = javacast("double",7);
ArrayAppend(poly,polyA);
ArrayAppend(poly,polyB);
ArrayAppend(poly,polyC);

You would then use this to call the  findWithin_Poly function.

hits = people.findWithin_Poly("loc",poly);

Now, I haven’t done this yet, but I would recommend doing it: add a utility method that would cast all of the elements to double. Simple and reduces clutter.

 

Unsetting Response Headers in an Apache Reverse Proxy Configuration When Serving PDF to Internet Explorer

Written by Sean Ryan on . Posted in App Dev

Yuck. IE.

Sometimes, we just need to suck it up and support IE even if it goes against everything we believe in. It’s well known that IE has a problem downloading PDFs over HTTPS when certain cache control headers are sent in the response.

See Microsoft’s own support site for this one: http://support.microsoft.com/kb/812935

The workaround for a developer is to make those headers disappear. There are several ways you can do this.

  1. In your application, don’t set them
  2. In your Web server, unset them.

Usually, the better approach is to configure your Web server to unset the header site-wide since the origin of the PDF behind your server is inconsequential. Removing them at the server level makes serving PDFs to all browsers work.

Unset Headers in Standard Apache Configuration

If you are using Apache in a standard way to front your application or site you can identify PDF requests using the Files directive.

<Files ~ \.pdf$>
   Header unset Cache-Control
   Header unset Pragma
</Files>

You should place this in the <VirtualHost *:443> section since this is a problem related only to serving over HTTP.

Unset Headers in Reverse Proxy Configuration

If you are using Apache as a reverse proxy and your PDFs are not on the filesystem relative to the site root, then you need to match the PDFs differently. In this case, you need to use a Location directive since the Files directive is used to match unproxied files.

<Location ~ \.pdf$>
   Header unset Cache-Control
   Header unset Pragma
 </Location>

Again, this should be placed in the <VirtualHost *:443> section.

Convert MySQL Database Character Encoding to UTF8

Written by Sean Ryan on . Posted in App Dev

Create a Database

To create a database that will default to UTF8 without modifying the server defaults run the following command

CREATE DATABASE dbName DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci

Broken down, it means:

CREATE DATABASE dbName – create a database named dbName (best name, ever)

DEFAULT CHARACTER SET utf8 – this database will use utf8 as its default character encoding. All tables created, will be configured to use utf8 for character columns.

DEFAULT COLLATE utf8_general_ci –  when comparing characters, ignore case (ci=case insensitive)

Convert an Existing Database

If you already have a database in some other character set, such as latin1, or ISO 8859-1, you can easily convert it to UTF8 by walking through a few simple steps.

1. Export your database

$> mysqldump -u -root -p \
       --complete-insert \
       --extended-insert \
       --default-character-set=utf8 \
       --single-transaction \
       --add-drop-table \
       --skip-set-charset \
 > dump.sql

This will include column names for your insert statements, efficient insert statements that don’t repeat column names, export in utf, not include setting the existing charset, and include drop table statements prior to create table statements.

2. Edit the Dump to Use UTF-8

cat dump.sql | \
    sed '/DEFAULT CHARACTER SET latin1/DEFAULT CHARACTER SET utf8/' | \
    sed '/DEFAULT CHARSET=latin1/DEFAULT CHARSET=utf8/'
  > dump-utf8.sql

This will just swap out latin1 for utf8. You can accomplish however you want to. I’m a fan of sed and awk and all things command line but the goal is clear; make latin1 stuff, utf8 stuff. Just be careful to qualify your  replacements. You don’t want to accidentally change data.

3. Drop Your Database

From the mysql prompt, run

> drop database dbName;

4. Import UTF8 Version

mysql -u root -p < dump-utf8.sql

 

That’s it. You could easily make a little shell script to do this and when need to convert a database from one encoding to another, just run it. Perhaps the command would look something like:

swap-encoding dbName fromEncoding toEncoding

How to Swap Out a Time Warner Modem With Your Own Faster One

Written by Sean Ryan on . Posted in App Dev

If you live in an area serviced by Time Warner Cable and you subscribe to high-speed Internet access, you’ve recently been mailed a notice with the infamous announcement of their plan to begin changing a “leasing” fee of $3.95/month for your cable modem. The mailing includes a way to avoid the fee by buying your own cable modem but with a caveat that doing so may leave you in dust when they “update” their network and your modem no longer supports their new network technology.

What they fail to mention is that over time, their equipment fails to give you the best experience and fastest speeds they have to offer but that you need to figure this out yourself and swap out your own equipment. When was the last time you ran some speed tests and researched Time Warner’s current network offerings [in your area] so you could decide if it was time to swap out your equipment ?

There are several points to consider when making this decision, so I’ll help lay it our for.

Cost

The cost of doing nothing is known outright: $3.95/month X 12 months = $47.40/year. Assuming they never ever increase that cost, you’re looking at about $50/year. I could shovel my driveway or I could have it plowed for about $50 so you really need to ask yourself, what it is that you value more. That being said, once I buy the modem, I never have to buy it again. Once I shovel my driveway, I’ll need to shovel it again 🙂 So let’s look at this a little differently. I’ve had Time Warner for 13 years now. If I was paying [an avoidable fee] all along, I would have donated $616.20 to Time Warner Cable by now.

Time Warner supplies you with a list of approved modems (http://www.timewarnercable.com/en/residential-home/support/topics/internet/buy-your-modem.html) and the prices range from around $50 to a little over $100. However, I found the SB6121 for $53 online even though it retails at $109.  A little sweat equity goes a long way sometimes.

How “Tech-Savy” Are You?

The process of doing all this is remarkably simple. I have to hand it to Time Warner on this one. I went through the process and the hardest part of all was picking out the modem – and even that was pretty simple after I understood what I needed.

Once you’ve selected your modem (see below) all you have to do is plug it in and move the Ethernet cable from your old one to the new one. Then, call Time Warner (1-866-321-2225) and say you need to activate a new modem. Trust me, they’re expecting your call. They know exactly what you need. They’ll ask one important question: “What’s the MAC address of your modem?” The MAC address, or Media Access Control number, is the way Time Warner authorizes your device to be on their network. To get this number, turn your modem over. It’s plastered right next to the serial number, model number, and all the other FCC stuff. If you run into any problem, Time Warner can and will help you through it.

Bandwidth

Unless you subscribed to Time Warner high-speed Internet in the last few months, it will almost certainly increase your bandwidth and consequently, your download speed – by a lot. Time Warner is in the process of upgrading their network to use the DOCSIS 3.0 protocol and without going into any detail, DOCSIS 3.0 is faster than DOCSIS 2.0, which most Time Warner Cable modems are able to handle. This boils down to you getting a faster download speed simply by using a modem that can handle it; without paying more. Yes, you could bring your current modem back and get one that has the DOCSIS 3.0 firmware on it, but you’d still be paying $3.95/month for it.

There are a lot of variables that determine network speed, such as which day and what time of that day but with a DOCSIS 2.0 (most Time Warner modems) and without turbo should give you about 10-15 Mb/second download and very close to 1Mb/second upload. With turbo, you’re looking at around 2 Mb/second upload.

When I swapped out their 2.0 modem with my own 3.0 modem, I saw my download speed jump from 15Mb/second to 32Mb/second! This test has been repeated dozens of times with similar results. Never has my download speed dropped below 29.

Other Considerations

I don’t want to get political but it’s worth a mention that the new policy of leasing the modem has the potential to be discontinued down the road. They’re already being sued because of it. “Send customers confusing notice of the fee in a junk mail postcard they’ll throw in the garbage, sock them with a $500 million dollar a year rate hike, then announce on your website that customer satisfaction is your No. 1 priority. That’s some way to deliver satisfaction.” That was said by one lawyer initiating a class action law suite on behalf of a New York and New Jersey client. (http://www.pcmag.com/article2/0,2817,2412196,00.asp)

For me, I see this as a way to take a little more control over my Internet access, increase my own speed, and avoid a little monthly fee. It’s not much but then again, buying a modem and calling Time Warner really doesn’t take all that much time.

What Model?

If you haven’t done so, check out the ist of modem they allow and what features they offer. I don’t need wireless access from my modem but you might. DOCSIS 3.0 was important to me and it should be to you too. Multiple ports was not important to me because I have a switch that goes to two routers. Spend ~$60 now or spend $600 over the next 13 years. The selection process really isn’t all that complicated so long as you’re realistic about your own needs.

 

Selecting Cloud Backup Software

Written by Sean Ryan on . Posted in App Dev

The need to backup data is nothing new but how we backup data does evolve and today we should all be at least exploring the cloud for a number of reasons; chief among them is that the cloud is offsite and maybe least among them is that all the cool kids are doing it. But, they are.

Recommendation

If you just want to know my recommendation for which cloud, it’s Amazon (http://aws.amazon.com/s3/). If you just want to know my recommendation for some client-side software, it’s CloudBerry (http://www.cloudberrylab.com/) for Windows and Arq (http://www.haystacksoftware.com/) for Mac users – both by a long shot. Read on to find out why.

Selecting The Cloud

I looked at several clouds for use beyond just backups and some for only backups. I kept coming back to the Amazon cloud because of its maturity, its massive economies of scale, its security model, the APIs, and its insane durability of eleven 9s – as they call it. The eleven 9s is interesting. It means that if you put a file up on the Amazon cloud, there is a .000000001% chance that file will be lost forever. Or, a 99.999999999% it will not be. They even offer a less durability level of 99.99% for about 25% of the cost if you want to “roll the dice.” Considering that until now, I’ve backup all my data on an external 1T drive, I’d be hard-pressed to estimate my own durability level at 90%. For one, the drive is roughly three feet from my computers (not offsite), for another it’s all on one drive (hope it doesn’t fail),  and not exactly the most physically secured device I have.

Let’s face it, if you can get over your trust issues with letting someone else watch over your [unreadable] data, then cloud storage hardly needs an argument. This article is about selecting how to PUT and GET your backup data and not how to choose a cloud so let’s move on…

Interacting with the Cloud

Depending on your nerd level, there may be several ways to interact with the cloud when using it for a backup store. With Amazon, you can build your own urls and use the RESTful services they provide – nerd level, awesome. Or, like a mortal, you can and should use a backup utility like Arq or CloudBerry that does the heavy lifting for you. There are many client utilities to choose from so I took a look at eleven of them and settled on two that stood far above the rest in almost every category of concern to me.

How To Select a Utility

There are plenty of aspects to look at when selecting a utility for your own backups but in my case, I narrowed it down to the ones that meant the most to me and sort of ordered them in my head and knew I would have to give some features up no matter what I found.

What’s the Cost?

Cost is obviously important. Almost all utilities have an upfront cost and most have transfer and storage fees. Some have monthly fees but they can usually be discounted if you sign up for 6 months, 1 year, or more at a time. Both Arq for Mac and CloudBerry for Windows have a one time cost of $29 for the software and you pay your cloud provider transfer and storage fees. $29 is pretty low when compared to the other nine popular solutions on my list.

Which Cloud?

Does the software support multiple clouds? This may not matter to you. Since I knew I wanted to use Amazon, I was only concerned with one cloud. However, not all utilities can use Amazon. For example, Carbonite uses their own cloud and there are no other options. Arq uses Amazon and CloudBerry support many clouds.

Does it deduplicate?

This is a very important feature and I refused to use a utility that was not capable of de-duplication. Simply, it means that if two files are found to have the exact same parts – such as an Excel chart embedded into multiple Microsoft Office Documents – then the duplicated part of those files is only backed up once. During a restore, the documents that shared the same content are restored properly as expected. When paying for transfer and storage, this reduction in backup sizes becomes a tremendous cost saver as your backup data grows. Both Arq and CloudBerry support this.

Does it Encrypt?

Even if the cloud provider encrypts your backup data, you may still want to encrypt the data yourself and prior to transfer. For example, Dropbox stores and encrypts your files on their own cloud but only after it gets to the cloud. If your data is interrupted on its way to the cloud, you’re information is exposed; even when using SSL. Moreover, Dropbox can technically un-encrypt your data – though they say the never do. If you encrypt your data prior to transmission, only you can decrypt it. In fact, if Dropbox un-encrypts your data (because remember, they always encrypt whatever you send to the cloud), all they’ll see is your encrypted version. This is exactly how 1Password syncing over Dropbox works. Arq and CloudBerry both offer encryption. CloudBerry actually offers a very wide range of options on this front.

How Much Space Do I Get and do I have and Control?

Unlimited space is what I needed but you may not need so much. This may depend on the tier you select with a provider. Carbonite, for example has an unlimited option for personal use but very limited to less limited and very expensive options for businesses. All tiers with Carbonite come with monthly or yearly fees that may be worth paying since there are no transfer or storage fees. If you end up with unlimited storage, you may want impose your own logical limits on the size of your storage to keep other costs down, such as long-term storage. Arq and CloudBerry both offer access to clouds with unlimited storage and both offer a way to keep your size within a designated budget. Very useful.

Does it Support Versioning?

Version is pretty cool. It’s a way to keep multiple copies of your files in case you need to restore a file before you overwrote changes and backup them up. Let’s say I backup some Apache configuration files and then make changes to them over the next several hours and by the end of the day, I realize I’ve messed up my http.conf file so badly, I need to start over but I never created a http.conf.orig file like a god little tinkerer. You can use your backup utility to find the original version even though you backed up up seven different version over the last seven hours (assuming you have your utility set to run once/hour). You’ll want to set a max version if possible, otherwise, your backups will grow very large in very little time. I keep mine for 30 days. Both Arq and CloudBerry support versioning from the cloud provider.

Can it be set to Run Automatically?

If you have an awesome backup strategy but forget to run it, what’s the point? I’m not a system administrator; I’m a software developer with no time to manage a system.  I rely heavily on automated tasks and calendar reminders. It was important to me to select a backup utility that could be set to run in the most flexible ways for me.  I have my utility set to run incremental backups once each hour on the hour. This recently proved invaluable to me and indirectly, to my company when I spilled an entire cup of coffee into my laptop. I was mad but lost zero bits of data because minutes before I spilled my coffee, the automated backup ran – and completed. Even if it had not run, the most I could ever be out is one hour’s work. Eating one billable hour isn’t as bad as eating 80 hours (10 days, the default for Time Machine). Both Arq and CloudBerry offer this functionality.

Can I Control its Network Usage?

You need to get your backup data to the cloud somehow and although some cloud providers offer a way for you to physically mail your hard drive to them, this isn’t a solution here 🙂 You’ll be transferring your files over the Internet and if you want to do this while you’re using the network, it would be very nice to have  way to control the network resources that your utility consumes. This is analogous to lowering a thread’s priority in a multi-threaded applications. If you plan to run your backups once per day at 2am while you’re sleeping, then this may not matter to you. Both Arq and CloudBerry offer network throttling.

Conclusion

For me, the choices were clear. For Windows user’s CloudBerry is a phenomenal solution. Not only does it offer everything I needed, it also offered everything I wanted. Like car insurance, I’ll be re-evaluating providers again down the road, but the next time, I’ll be using CloudBerry’s many features as a benchmark when comparing other utilities. My Mac has been running Arq for several weeks now without a problem and I’ve made several individual file restores without any complications.

Cloud backups are the way to go, but be sure to chose an awesome utility to make your life easier.

 

Make Eclipse Editor Work with Retina Displays

Written by Sean Ryan on . Posted in Software

I recently had to buy a new computer – a MacBook – because I spilled a full cup of coffee all over the keyboard of my 17″ MacBook.  I was a little upset. Apple stopped making 17″ laptops so to make up the for the lack of screen real estate, I decided to go all out with a shiny new retina display. I admit, I’m more impressed than I thought I would be but the technology comes at a cost.

Not all applications look nice with the display. With retina, applications need to display different icons and fonts that are made specifically for a much higher pixel density. Any web developer already deals with this when targeting iPhones or iPads, but since very few non-mobile devices have such displays, not all desktop applications have the necessary files yet.

Eclipse is such a popular IDE that I was devastated to see it doesn’t support retina yet. I use the IDE all day long so I scoured the Internet tubes for a solution and found one that is good enough. It only updates the fonts for the editor part and not any of the graphics – obviously – but that’s good enough for now. Here’s what you need to do…

  1. Right click and  “Show package contents” on the Eclipse.app. (STS.app, if using Spring Source Toolkit)
  2. Edit Contents/Info.plist. Just above the closing </dict> </plist>, add <key>NSHighResolutionCapable</key> <true/>
  3. Make a copy of the app (Eclipse copy.app)
  4. Open the copy
  5. Close the copy
  6. Rename the original (Eclipse-NON-RETINA.app)
  7. Rename the copy (Eclipse.app)
  8. Open
  9. Enjoy