the next door (so called) geek @rakkimk | your next door geek | friend | blogs mostly on technology, and gadgets.

Windows Azure Websites - WebJob to upload FREB logs to Azure Blob Storage

Currently in Windows Azure Web sites, there is an option to store your website logs to Azure Blob Storage, but however, the FREB logs – failed request tracing logs, can only be stored in the file system. You will then grab them via your favorite FTP tool, and do the analysis. One of my co-worker asked this question on if we can store the Failed Request Tracing logs in the Azure Blob Storage, and Richard Marr gave this interesting idea of using WebJobs to move the files to Azure Blob Storage. I tried this quickly, and have a beta version of a similar webjob ready for you if you want to try it. I might revive the code sometime later when I find time, but feel free to use the code that I’ve posted in this GitHub repo.

If you notice, I’ve not used the WebJobs SDK as such, but as a normal C# program that monitors the folders and uploads the files created in that folder to Azure Blob Storage. It creates a container called “freblogs” if it doesn’t exist, and stores the files there. You can modify the code to cater to your need.

Entire code of my small C# application:

using System;
using System.IO;
using System.Configuration;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Auth;
using Microsoft.WindowsAzure.Storage.Blob;
 
namespace PushToStorage
{
class AzureStorageHelper
    {
        CloudStorageAccount storageAccount;
        CloudBlobClient blobClient;
        CloudBlobContainer container;
        CloudBlockBlob blockBlob;
bool deleteAfterUpload = false;
public AzureStorageHelper()
        {
 // Retrieve storage account from connection string.
            storageAccount = CloudStorageAccount.Parse(ConfigurationManager.ConnectionStrings["StorageConnectionString"].ConnectionString);
 
 // Create the blob client.
            blobClient = storageAccount.CreateCloudBlobClient();
 
 // Retrieve a reference to a container. 
            container = blobClient.GetContainerReference("freblogs");
 
 // Create the container if it doesn't already exist.
            container.CreateIfNotExists();
 string tmp = ConfigurationManager.AppSettings["DeleteAfterUpload"];
 if (tmp != null)
            {
                deleteAfterUpload = (Int32.Parse(tmp) == 1) ? true : false;
            }
 
        }
 
public void UploadFileToBlob(string name, string path)
        {
 try
            {
                Console.WriteLine("Starting uploading " + name);
 using (var fileStream = System.IO.File.OpenRead(path))
                {
                    blockBlob = container.GetBlockBlobReference(name);
                    blockBlob.UploadFromStream(fileStream);
                    Console.WriteLine(name + " successfully uploaded!");
 if (deleteAfterUpload)
                    {
                        fileStream.Close();
                        File.Delete(path);
                        Console.WriteLine(path + " deleted!");
                    }
                }
            }
 catch (Exception ee)
            {
                Console.WriteLine(ee.Message);
            }
        }
 
public bool IsFileReady(String sFilename)
        {
 try
            {
 using (FileStream fileStream = File.Open(sFilename, FileMode.Open, FileAccess.Read, FileShare.None))
                {
 if (fileStream.Length > 0)
 return true;
 else
 return false;
                }
            }
 catch (Exception)
            {
 return false;
            }
        }
    }
class Program
    {
static AzureStorageHelper azStorageHelper;
static void Main(string[] args)
        {
            Console.WriteLine("Initializing AzureStorageHelper!");
            azStorageHelper = new AzureStorageHelper();
 
 string path = ConfigurationManager.AppSettings["directory"];
 
 string[] directories = Directory.GetDirectories(path, "*W3SVC*");
 
            FileSystemWatcher[] fsw = new FileSystemWatcher[directories.Length];
            Console.WriteLine(path + " " + fsw.Length);
 for (int i = 0; i < directories.Length; i++)
            {
                fsw[i] = new FileSystemWatcher(directories[i]);
                fsw[i].NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.LastWrite | NotifyFilters.FileName | NotifyFilters.DirectoryName | NotifyFilters.Size;
                fsw[i].Created += fsw_Created;
                fsw[i].IncludeSubdirectories = true;
                fsw[i].EnableRaisingEvents = true;
                Console.WriteLine(String.Format("{0} Started watching directory {1} for files!", DateTime.Now.ToString(), directories[i]));
            }
 
            Console.ReadLine();
            Console.WriteLine(DateTime.Now.ToString() + " Stopping!");
        }
 
static void fsw_Created(object sender, FileSystemEventArgs e)
        {
            FileInfo fileInfo = new FileInfo(e.FullPath);
 while (!azStorageHelper.IsFileReady(e.FullPath))
                System.Threading.Thread.Sleep(1000);
 
            Console.WriteLine(" Created " + e.Name + " " + e.FullPath);
            azStorageHelper.UploadFileToBlob(e.Name, e.FullPath);
        }
    }
}

As you see, the code reads from web.config for the Azure Blob Storage Connection String, as well as two other settings – folder to watch, and option to delete the file after uploading. They are self explanatory, so you can use this to move any files of your folders to the Azure Blob Storage. You need to install the Windows Azure Storage Nuget Package. Once you have finished the application as a standalone, you need to create a compressed zip file consisting of your .exe, .exe.config, and other Windows Azure Storage reference DLL (since these are not part of WAWS instance by default), and follow the steps in this article to create a WebJob. You can choose to create different type of the job as per your need.

Articles below helped me writing this small tool:

How to use the Windows Azure Blob Storage Service in .NET
http://www.windowsazure.com/en-us/documentation/articles/storage-dotnet-how-to-use-blobs-20/#upload-blob

Using the WebJobs feature of Windows Azure Web Sites

http://curah.microsoft.com/52143/using-the-webjobs-feature-of-windows-azure-web-sites

Hope this helps!

Pingbacks and trackbacks (5)+

Add comment

biuquote
  • Comment
  • Preview
Loading