the next door (so called) geek @rakkimk | your next door geek | friend | blogs mostly on technology, and gadgets.

WAWS - WebJob to upload FREB files to Azure Storage using the WebJobs SDK

After writing my earlier post on creating a simple WebJob to upload the Failed Request Tracing logs automatically to Windows Azure Blob Storage account, I was discussing this with the awesome development team of the WebJob SDK, Amit Apple, and Mike Stall. And, the outcome is, getting my sample modified to use the awesome WebJobs SDK that eases a lot of tasks. And there is more to it – cool Azure Jobs Dashboard with your Windows Azure Web sites giving you a cool dashboard of your WebJobs messages getting processed.

With the WebJobs SDK, there is automatic way of calling certain functions. You can check Scott’s blog where he have used a function that just monitors his Azure Blob Storage account for an new blob to be created, and process that image, and push it to his Azure Blob Storage account itself. The code he has written is very less, just one function, and wrapping that inside an application with the WebJobs SDK. If you notice his function, he used attributes [BlobInput], and [BlobOutput]. However, in my case, for the example that I was trying – to push the files from file system to the Azure Blob storage, I need some thing like [FileInput] which isn’t available, but WebJobs SDK seems to give custom ways to hook in functions to help with the interaction with the Azure Blob Storage account. I’ve modified my function that uploads the file as below. I also call this Upload function from another function.

public static void Upload(string name, string path, // Local file 
                                    [BlobOutput("freblogs/{name}")] Stream output,
 bool deleteAfterUpload)
        {
 using (var fileStream = System.IO.File.OpenRead(path))
            {
                fileStream.CopyTo(output);
            }
 
 if (deleteAfterUpload)
            {
                File.Delete(path);
            }
        }
public void UploadFileToBlob(string name, string path)
        {
            var method = typeof(AzureStorageHelper).GetMethod("Upload");
            _host.Call(method, new { name = name, path = path, deleteAfterUpload = deleteAfterUpload });
        }

If you notice, from the UploadFileToBlob() which I’m calling from my FileSystemWatcher callback, I’m not doing any single bit of code to upload the blob to the Azure Blob Storage, I just call another function via the _host (of type Jobhost), and pass the parameter name, path, and the boolean, WebJobs SDK function automatically fills in the Stream output there which would be created as a blob under the “freblogs” container, with the same name that I pass into this function, “name”. You just need to configure the connection string with name “AzureJobsData” for the application in it’s app.config file. Pretty awesome, isn’t it? This WebJobs SDK is in alpha I’m told, so I’m really waiting to see what are the new features in the final version. Sure, the team has set a high bar for themselves Smile

If this isn’t enough, you have an awesome site extension for your Azure Websites that shows all these WebJobs operations with the Azure Blob Storage. I was trying to understand how this works, what WebJobs SDK does is, create another container called “azure-jobs-invoke-log” in your Blob storage account, and stores the logs inside, which then are fetched by the AzureJobs site extension, and shown to you. Here is what my Storage Account shows the containers, the “freblogs” that contain all the FREB files, and the “azure-jobs-invoke-log” showing the container that holds all the log messages of WebJobs SDK.

image

And, to enable the Site Extension, you need to make sure you first configure the connection string named “AzureJobsRuntime” having the same connection string to the Blob Container.

image

After saving this connecting string, then you have to go to the URL https://<yoursitename>.scm.azurewebsites.net/AzureJobs URL. This page shows you details about your WebJobs configured on the Blob Storage account that you have configured, and it’s full details. You can click on each invocation to see it’s details. Remember, in my case it is a custom function that I’m invoking, so it will show you all the details about the parameters that were passed in, and the result of that call. Also, it has a link to the file that we just uploaded – you can download that file by just clicking on the output hyperlink in the 2nd screenshot below.

image

image

Pretty awesome! Don’t wait. Add more power, and background processing to your Azure Websites using the new WebJobs SDK.

Windows Azure Websites - WebJob to upload FREB logs to Azure Blob Storage

Currently in Windows Azure Web sites, there is an option to store your website logs to Azure Blob Storage, but however, the FREB logs – failed request tracing logs, can only be stored in the file system. You will then grab them via your favorite FTP tool, and do the analysis. One of my co-worker asked this question on if we can store the Failed Request Tracing logs in the Azure Blob Storage, and Richard Marr gave this interesting idea of using WebJobs to move the files to Azure Blob Storage. I tried this quickly, and have a beta version of a similar webjob ready for you if you want to try it. I might revive the code sometime later when I find time, but feel free to use the code that I’ve posted in this GitHub repo.

If you notice, I’ve not used the WebJobs SDK as such, but as a normal C# program that monitors the folders and uploads the files created in that folder to Azure Blob Storage. It creates a container called “freblogs” if it doesn’t exist, and stores the files there. You can modify the code to cater to your need.

Entire code of my small C# application:

using System;
using System.IO;
using System.Configuration;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Auth;
using Microsoft.WindowsAzure.Storage.Blob;
 
namespace PushToStorage
{
class AzureStorageHelper
    {
        CloudStorageAccount storageAccount;
        CloudBlobClient blobClient;
        CloudBlobContainer container;
        CloudBlockBlob blockBlob;
bool deleteAfterUpload = false;
public AzureStorageHelper()
        {
 // Retrieve storage account from connection string.
            storageAccount = CloudStorageAccount.Parse(ConfigurationManager.ConnectionStrings["StorageConnectionString"].ConnectionString);
 
 // Create the blob client.
            blobClient = storageAccount.CreateCloudBlobClient();
 
 // Retrieve a reference to a container. 
            container = blobClient.GetContainerReference("freblogs");
 
 // Create the container if it doesn't already exist.
            container.CreateIfNotExists();
 string tmp = ConfigurationManager.AppSettings["DeleteAfterUpload"];
 if (tmp != null)
            {
                deleteAfterUpload = (Int32.Parse(tmp) == 1) ? true : false;
            }
 
        }
 
public void UploadFileToBlob(string name, string path)
        {
 try
            {
                Console.WriteLine("Starting uploading " + name);
 using (var fileStream = System.IO.File.OpenRead(path))
                {
                    blockBlob = container.GetBlockBlobReference(name);
                    blockBlob.UploadFromStream(fileStream);
                    Console.WriteLine(name + " successfully uploaded!");
 if (deleteAfterUpload)
                    {
                        fileStream.Close();
                        File.Delete(path);
                        Console.WriteLine(path + " deleted!");
                    }
                }
            }
 catch (Exception ee)
            {
                Console.WriteLine(ee.Message);
            }
        }
 
public bool IsFileReady(String sFilename)
        {
 try
            {
 using (FileStream fileStream = File.Open(sFilename, FileMode.Open, FileAccess.Read, FileShare.None))
                {
 if (fileStream.Length > 0)
 return true;
 else
 return false;
                }
            }
 catch (Exception)
            {
 return false;
            }
        }
    }
class Program
    {
static AzureStorageHelper azStorageHelper;
static void Main(string[] args)
        {
            Console.WriteLine("Initializing AzureStorageHelper!");
            azStorageHelper = new AzureStorageHelper();
 
 string path = ConfigurationManager.AppSettings["directory"];
 
 string[] directories = Directory.GetDirectories(path, "*W3SVC*");
 
            FileSystemWatcher[] fsw = new FileSystemWatcher[directories.Length];
            Console.WriteLine(path + " " + fsw.Length);
 for (int i = 0; i < directories.Length; i++)
            {
                fsw[i] = new FileSystemWatcher(directories[i]);
                fsw[i].NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.LastWrite | NotifyFilters.FileName | NotifyFilters.DirectoryName | NotifyFilters.Size;
                fsw[i].Created += fsw_Created;
                fsw[i].IncludeSubdirectories = true;
                fsw[i].EnableRaisingEvents = true;
                Console.WriteLine(String.Format("{0} Started watching directory {1} for files!", DateTime.Now.ToString(), directories[i]));
            }
 
            Console.ReadLine();
            Console.WriteLine(DateTime.Now.ToString() + " Stopping!");
        }
 
static void fsw_Created(object sender, FileSystemEventArgs e)
        {
            FileInfo fileInfo = new FileInfo(e.FullPath);
 while (!azStorageHelper.IsFileReady(e.FullPath))
                System.Threading.Thread.Sleep(1000);
 
            Console.WriteLine(" Created " + e.Name + " " + e.FullPath);
            azStorageHelper.UploadFileToBlob(e.Name, e.FullPath);
        }
    }
}

As you see, the code reads from web.config for the Azure Blob Storage Connection String, as well as two other settings – folder to watch, and option to delete the file after uploading. They are self explanatory, so you can use this to move any files of your folders to the Azure Blob Storage. You need to install the Windows Azure Storage Nuget Package. Once you have finished the application as a standalone, you need to create a compressed zip file consisting of your .exe, .exe.config, and other Windows Azure Storage reference DLL (since these are not part of WAWS instance by default), and follow the steps in this article to create a WebJob. You can choose to create different type of the job as per your need.

Articles below helped me writing this small tool:

How to use the Windows Azure Blob Storage Service in .NET
http://www.windowsazure.com/en-us/documentation/articles/storage-dotnet-how-to-use-blobs-20/#upload-blob

Using the WebJobs feature of Windows Azure Web Sites

http://curah.microsoft.com/52143/using-the-webjobs-feature-of-windows-azure-web-sites

Hope this helps!