Node.js Tools for Visual Studio: Deployment Hiccups

Next steps sometimes can be cumbersome. Yesterday, I was trying to deploy a node.js application with express.js to an Azure Website. I was following this tutorial. I figured I’d never make it on first attempt.

Much to my surprise, I did in just six minutes.

On a personal note, I’d like to add that publishing to Azure Websites using git (and getting this really useful deployment history) is something that makes me happy on every single deployment.

So today I was ready for the next step. I am usually working with Visual Studio, so I wanted Node.js Tools for Visual Studio. This give me a lot of benefit out of the box that I’m not even going to mention here, but Scott Hanselman was pretty excited about it some time ago.

So after installing I created a blank Node.js app. To be precise, it was this option.

NodeToolsProject

I was not using any of the Azure variants listed here, as I did not need the deployment scripts they add in this case. For a difference overview of these project types, check out the “Converting to an Azure project type” section on their deployment wiki page.

After the project wizard did it’s job, I created a new commit, created an azure website, added its git endpoint as remote, and pushed. After yesterday’s experience, I figured it would work in about 30 seconds.

Much to my surprise, it didn’t. Not even in six minutes.

Turns out there were two problems.

Problem 1

The project wizard creates a .gitignore file. This seems to be a variation of the standard Visual Studio .gitignore file, which excludes everything beneath bin folders. However, for some reason the project template uses a www file instead of server.js. Guess its path? Right, bin/www. Which means it gets excluded, so I had to make sure to explicitly add it.

Problem 2

Are we good now? Not quite. Contrary to the tutorial I followed yesterday, my “real code” does not live in the root of my git repo. Instead, that’s where the .sln file lives, the project is one level deeper.

| root
| – mynodeapp.sln
| – mynodeapp/
| – mynodeapp.njsproj
| – app.js
| – package.json
| etc …

Which means, Azure will get confused about the nature of your project (won’t recognize it as node.js application) and fail more or less miserably. It just didn’t tell me, as deployment history showed green.

The second last paragraph on this Kudu help site brought me on the right track, as well as a comment on the first tutorial mentioned above: it is necessary to tell Azure website to look one level deeper. The most flexible way of doing so is an app setting entry in the Azure Management Portal itself, under the Configure tab of your website.

The key needs to be Project, and the value has to be <projectfolder>.
Notice the dot at the end. Like so:

NodeWebsiteAppSetting

Saved the settings, headed over to the deployment history and forced a redeploy of my last commit.

And sure enough, it was running as expected.

Hope that helps

Advertisements

Azure Websites with NuGet Packages From A Custom or Private Feed

Azure Websites offer the capability to deploy from source control, such as Bitbucket or GitHub, instead of you building a deployment package or doing a web publish.

The way that works is that Bitbucket, for example, will send a push notification to your Azure Website whenever a commit is being pushed to it. This will make Azure pull the missing changesets and trigger a build. To make this build possible, it performs a NuGet package restore (if your solution is configured in such a way, that is).

 

This way of deploying has the added benefit of a build service test for your repository: If you’ve forgotten to check in relevant files, your deployment on Azure Websites will fail because the build will break. You don’t get this verification when building a deployment package with files present in your local workspace.

 

While this is a pretty handy and fast way of deploying, it does not work out of the box if you’re using NuGet package feeds that are not accessible for that build service.

There are some possible solutions for that:

  1. Host your NuGet packages in a publicly available location
  2. Direct link these assemblies with copylocal=true instead of using the package structure
  3. Copy them using FTP

You might not want to host your custom packages publicly (1) or circumvent the entire idea of NuGet in the first place just to enable this source code deployment scenario (2)

At Tekaris, we are using custom NuGet packages in a private feed hosted within our own infrastructure, so we chose option 3. Of course, it helps that these packages seldom change.

 

How’s this working? First, go to the management portal at https://manage.windowsazure.com and navigate to your website’s quick start page.

image

Then download the publish profile. Open this file in a text editor and look for FTP host, user and password information.

Use this information to open a FTP connection and navigate to /site/repository/packages

Now simply copy all these missing packages from your local workspace (where they have been used for building your app) to this package directory. Azure will not clean this directory when you do a new commit/deploy, so as long as you don’t need newer versions of them, this is a one-time operation.

 

Hope that helps,

Jonas

Visiting the WCF Data Services Toolkit April 2011 Update

Looking for a way to expose data from Windows Azure Table Storage via an OData webservice I quickly discovered that in its current form, WCF Data Services are primarily meant to expose relational data when using the Entity Framework. Since working with Azure Storage – such as TableStorage – typically does not involve a relational data model, OData seemed to be a difficult path. Not using Entity Framework means you will have to implement your own LINQ provider to get the job done. This is not a trivial task as it involves juggling – or rather messing – around with expression trees.

At this point I discovered the WCF Data Services Toolkit on Codeplex which circumvents the need to write your own LINQ provider by using a repository pattern architecture for you to hook into. I also found documentation to get me started, however I discovered later that the documentation does not match the current (beta) release of the toolkit.

As a prerequisite, please read the author’s blog post about updating resources with the toolkit. Here he describes the IWriteableRepository interface. Please note that this documentation is partly out of date as with the april update:

 

IWriteableRepository interface is gone

But fear not. It is a breaking change, but simply removing it from the class declaration will do the trick. This is because methods on your repository will be called by convention (hence by reflection). This has multiple advantages which the author mentions at the end of the updating resources post.

CreateDefaultEntity has been renamed to CreateResource

While the previous change might have been obvious, this one is not. But again, this is easy to solve, just rename the method.

 

This is it for the changes. Some random things I discovered while using the Toolkit:

Make sure to implement CreateResource. In there, simply new up a new instance of your entity (or do more advanced stuff if necessary). Otherwise, the toolkit will try to use Activator.CreateInstance() to create an entity instance, which did not work for me because the type to create an instance from was in a different assembly. This is a known issue (#12 on Codeplex) and I sent a patch for this to the author.

 

Just for reference: The list of methods one can implement in the repository:

  1. public void Save(<YourType> entity)
  2. public void Remove(<YourType> entity)
  3. public void CreateRelation(<YourType> entity, <YourOtherType> relatedEntity)
  4. public object CreateResource()

So far, I really like the toolkit and how it helped me to work with Azure Storage. Keep up the good work!

New Windows Azure Feature Announced at PDC 2010

This morning, I received a newsletter from the Windows Azure Platform Team. Besides the fact that it’s formatting is horrible as usual since all German umlauts are displayed as some weird unreadable symbol, it contained a lot of new and exciting features for Windows Azure. Also be sure to check out this video which provides a fast overview of new features for Windows Azure.

Extra Small Instances

This is one of the features I was really waiting for. Before this announcement, the smallest instance size was Small, having a price tag of $0.12 per hour. Now there is an Extra Small instance size with $0.05 / hour.

I went ahead and replicated the table from this Microsoft web site to have the most current overview on my blog. This should come in handy, at least for me because I will keep forgetting this stuff.

Compute Instance Size CPU Memory Instance Storage I/O Performance Cost per hour
Extra Small 1.0 GHz 768 MB 20 GB Low $0.05
Small 1.6 GHz 1.75 GB 225 GB Moderate $0.12
Medium 2 x 1.6 GHz 3.5 GB 490 GB High $0.24
Large 4 x 1.6 GHz 7 GB 1,000 GB High $0.24
Extra large 8 x 1.6 GHz 14 GB 2,040 GB High $0.96

As explained in this video, Extra Small Instances share resources (CPU, memory) with other VMs on the same node. Furthermore, the network bandwidth is capped at around 5 Mbps. This is not the case with larger instance sizes, where CPU and memory are not shared and your service can leverage unused bandwidth.

I believe this is a very good idea. Amazon’s smallest Windows instance (Micro Instance) is available at a cut-price $0.03 per hour, but we can’t compare these offerings 1:1 because Azure offers more functionality. We don’t have to pay extra for load balancing services and we don’t have to worry about OS updates etc. Besides, Azure’s Extra Small instance has 155MB more RAM, which could make a lot of difference.

This is why I like this new instance size so much: It means that it just got easier to get a small service running without downtime. By that I mean that I can now pay 2 extra small instances for a small service which gives me the ability to enable rolling upgrades without downtime. This includes automatic OS updates as well as my own updates to my service when delivering new features. This is still cheaper than having one small instance without all these benefits. I would expect this to activate the SLAs also, but I’m not sure about that. This won’t be the case while this feature is still in beta.

Things I Believe Windows Azure is Missing

Lately I was in the comfortable situation of having some time to dig deeper into some features of Windows Azure. While I am really excited about what Azure means to me as a .NET developer who wants to leverage PaaS and not IaaS, I also feel that some very important features are still missing.

Some missing features require workarounds, others are simply showstoppers. For example, this week I hit a wall with Table Storage. Ouch. I was really happy about the approach I used with it and had even already applied some workarounds to my solution when, suddenly and sadly, I had to realize that Table Storage just doesn’t support case insensitive search.

Without going into much detail, let me put all this into a list. While it may seem that this post is essentially a rant about Windows Azure, that is not true. The rant is really only about documentation, see below. Everything else are just features I want to have and I believe the Azure platform needs to have ASAP if MS doesn’t want to get left behind. Some other big players in Cloud Computing are doing a splendid job of throwing new features at developers almost monthly and letting them know as soon as possible.

So all this boils down to: Deliver crucial new features. Do it fast. Let everyone know about it.

Table Storage

There is no support for secondary indexes. Amazon SimpleDB has it. Missing this kind of feature means we will have to store information redundantly, just to be able to query efficiently for them. This results in additional development effort as well as increased storage costs. Since storage is cheap, this is ok. Since when done properly, Table Storage can do an enormous scale out, I am even willing to invest some more time to get this right. So this is not necessarily a showstopper.

There is no support for case insensitive search. This means that I would have to store a lowercase version of whatever I need to search for, together with the version containing the correct spelling.

There is also no way for doing full text searches on the storage. There is a workaround for StartsWith() but none for EndsWith() or Contains(). And I’m not talking about not being able to do that efficiently. I’m talking about not being able to do that. Because the LINQ provider just doesn’t support these kinds of queries. And no, doing that on the client side is not always an option.

So MS, please, don’t tell me that Table Storage is missing all those features by design and that it was always intended to be only a very simple way of storing data. Just don’t. Give it a few more features and make it a powerful No-SQL storage. I’m not asking for making it support all the relational stuff like cascading deletes and the like. I understand that it is non-relational. But don’t stop on half the way. Continue, quickly.

Hosting

Yes I know this one is probably just for devs trying to grok the way Azure works or maybe small startups, but still … hosting a small service on Azure is just too expensive. As already mentioned, this is not a problem when you put your virtual machine’s power to good use, but the pay-as-you-grow model has a flaw here – the inital price might be too high if your needs are too low. Ah yes, Amazon has that, too.

Sending Emails

Did I mention that Amazon has a feature called SNS? This is not sufficient for sending emails to your customers, but it is perfectly acceptable for anything else, including email messaging in case of failure etc. There are solutions for that, but these are merely workarounds.

Caching

While I am happy that there are projects like CloudCache that are based on Memcached, I really hope that support for Velocity a.k.a. AppFabric Cache is coming soon. And by soon I do not mean PDC 2011. If there is no announcement (with specific date) at PDC 2010, I will go with Memcached.

Fixed IPs

IPs seem to remain fixed over the lifetime of a deployment. While this may be true, there is no official statement from Microsoft saying that this behavior is supported. To some, that might seem fair enough since you don’t have to bother about that anyway, right? Well, say you’d have to communicate with some other service that requires your IP to be whitelisted. Not your domain but your IP. Then assume that it will take some time to get another whitelist entry in case something happens to your first IP. You could try to get a ‘backup IP’ whitelisted parallel to your real IP, but this would require another deployment to run on a different production slot, so this is also a clumsy workaround.

And Amazon has the concept of – ah, never mind.

Documentation

Honestly. This is just too hard. Its not surprising the Azure community keeps on providing link lists and compilations of Azure resources. Information is scattered, insufficient and sometimes out-dated.

And I’m getting tired of saying this, but compared to Microsoft’s approach, Amazon’s is really centralized. You find a long list of documentation for every feature AWS offers on the starting page for that feature and from there on, you can follow links to complementary information.

There is a monthly AWS newsletter saying “we are happy to announce”. Well, I am happy if there is ANY way for me to figure out what Microsoft is up to in the next few months with regard to Azure. Why is communication so bad on this topic?

Last time I checked, the official voting forum for Windows Azure Feature Requests lists 8 completed requests and 1 accepted request (secondary indexes for Table Storage). Even then, there has been no further information on secondary indexes since like 11 months. No, let me check that again, this can’t be true … aargg, it is true. There are a lot of other very important requests but none of them made it to ‘accepted’. Does this mean the Azure team is not currently working on any of these feature requests? I hope not! Really, I hope not.

I would like to see that Microsoft uses this platform (or any other means, just DO it) to communicate upcoming changes to developers and decision makers. That way, everybody would be able to plan ahead and be less frustrated. Developers like me would have a central way of being kept in the loop and appreciate new features instead of being angry for not being told.

Only they don’t. So they won’t. So they are not. And we haven’t. So I am not (at least to a certain degree).

And I definitely believe it is not a good idea for MS to keep roadmaps and timelines as secret as possible, wait a whole year until next PDC and only then make information about all that publicly available. That being said, I really hope there will be some interesting announcements at PDC 2010. I need certain new features to make it easier for me to get customers adopt Azure. Otherwise they will rather go with Amazon.

How to Setup AWS Tools and Unmonitor EC2 Instances

As paying Amazon AWS Customer I recently visited the AWS Console and activated the CloudWatch service for a specific instance. I was happy to see that it almost immediately started to collect data.

I felt like some important stats were missing, but this is not the point of this post.

I also felt like an important option was missing, and that is precisely the point of this post.

The problem was: There is no way to stop monitoring an instance through AWS Console. This makes perfect sense from Amazon’s point of view since this is the way to make (a little bit of) money.

AWS Command Line tools to the rescue

I downloaded and set up the tools and then was able to issue a request to AWS stating that monitoring of this specific instance had to stop.

Here’s how you do it, it should take around 10 minutes from zero to issuing the request.

  1. Download the AWS Command Line Tools
  2. Set up the Command Line Tools, which basically means
    1. Set environment variables to the tools (EC2_HOME), to your AWS keys and certificates(EC2_PRIVATE_KEY and EC2_CERT), and probably also to Java (JAVA_HOME)
    2. Specify which Region you want to use by setting the EC2_URL variable like so (on Windows):
      set EC2_URL=https://ec2.eu-west-1.amazonaws.com
  3. And then you’re good to go and have a look into the Command Line Reference and issue the request to AWS. In this case, I wanted to stop monitoring an instance, which is then as simple as going into the \bin directory of the AWS tools and issue
    ec2-unmonitor-instances <instanceId>

Azure Launch Day in Stuttgart Wrap Up

After having listened to Tim Fischer et al, the summary of this conference manifested in my mind like this:

Microsoft still hasn’t decided how to pronounce Azure. Which makes kind of sense since GB and US pronunciations seem to differ quite a bit.

Some other topics were of interest, too 😉 I’d like to give a brief summary here because the conference served as a trigger for me to revisit the latest evolution in Microsoft’s cloud computing.

Different VM Sizes

Azure now lets you choose how much power your hosting instances are sporting. There are four different sizes available:

Name Price / hour CPU RAM Instance Storage
Small $0.12 1 x 1.6 GHz 1.75 GB 250 GB
Medium $0.24 2 x 1.6 GHz 3.5 GB 500 GB
Large $0.48 4 x 1.6 GHz 7 GB 1,000 GB
X Large $0.96 8 x 1.6 GHz 14 GB 2,000 GB

Pricing and features is almost equal to Amazon Web Service Standard Instances. For details, see their different types and pricing.

Upcoming Features

There will be blobs that can be mounted as NTFS drives called XDrives. As Ray Ozzie said:

Perhaps most significantly is in a new storage type that we call XDrive. Azure XDrives are Azure storage blobs that are mountable as regular NTFS volumes, a drive mapped, cached, durable volume, accessible by mapping an Azure page blob into an NTFS VHD.

He also announced that the Azure portfolio will have a feature that is really more an IaaS feature than a PaaS feature:

As we move forward and think about ways we can simplify being able to take the investments that you’ve made in the Windows Server environment and move them into Windows Azure, one of the things that we’re doing is allowing you to create your own image. We will do this next year. This is another feature that’ll come in 2010. We’ll allow you to create your own image, which has all of the software configured exactly the way you want it.

As far as I know, this feature is called “Virtual Machine Role” but no one knows. Maybe even Microsoft doesn’t know. And if the do know, they won’t pronounce it. Hell no.

I also heard that in 2010 Worker Roles can be addressed directly from the web without having to route traffic through Web Roles. Didn’t quite understand why there are 2 different roles, then.

Blobs

Blobs are really getting useful. I already knew they had the ability to be public or private, but these two new features were news to me:

  • By specifying HTTP Cache-Control policy for each blob, Azure Blob Storage can be used as Content Delivery Network
  • Snapshots of blobs can be taken to create read-only backups

Pricing Options

As we were told, there will be different pricing options. One of these options is useful for systems that already have a certain level of consumption and want a better pricing strategy for that compared with the very flexible but relatively costly “Pay as you grow” strategy. The first is more like Amazon AWS reserved instances.

And BizSpark Members will get Azure Hosting and SQL Azure for free for 8 months (don’t know details yet).

Misc

Summary

Compared to Amazon, I like the idea of PaaS (Azure being the first choice for a .NET developer like me). When I want give one of my ideas a try and build a web application, I surely don’t want to care about all this tedious infrastructure stuff like firewall, backups, load balancing, security updates etc.

It’s interesting to see that Microsoft is announcing a move more towards IaaS that early. This seems to be driven by early customer feedback. There must be a need for more flexible environments and they don’t want to lose those people to Amazon.

I really dig some of the new features. Good job so far, keep it coming. Looking forward to the EU datacenters.

More links:

Windows Azure Storage at PDC 2009

PDC 2009 (German Blog)