<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Cloudronics - Medium]]></title>
        <description><![CDATA[Curated articles for Developers and Enterpreneurs - Medium]]></description>
        <link>https://medium.com/cloudronics?source=rss----63dd8998b468---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Cloudronics - Medium</title>
            <link>https://medium.com/cloudronics?source=rss----63dd8998b468---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 03:42:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/cloudronics" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[Hyperledger fabric blockchain cluster without using docker containers]]></title>
            <link>https://medium.com/cloudronics/hyperledger-fabric-blockchain-cluster-without-using-docker-containers-746359e174c?source=rss----63dd8998b468---4</link>
            <guid isPermaLink="false">https://medium.com/p/746359e174c</guid>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[hyperledger]]></category>
            <category><![CDATA[hyperledger-fabric]]></category>
            <category><![CDATA[systemd]]></category>
            <category><![CDATA[system-administration]]></category>
            <dc:creator><![CDATA[anoop vijayan maniankara]]></dc:creator>
            <pubDate>Wed, 11 Dec 2019 09:06:34 GMT</pubDate>
            <atom:updated>2019-12-19T21:47:14.593Z</atom:updated>
            <content:encoded><![CDATA[<h3>Setting up a hyperledger fabric blockchain cluster without using docker containers</h3><p>This article was originally published by me in <a href="https://medium.com/u/816378d6a94d">UpCloud</a>’s <a href="https://upcloud.com/community/tutorials/hyperledger-fabric-systemd/">blog</a>.</p><p>Hyperledger Fabric is an enterprise-grade permissioned distributed ledger framework for developing solutions and applications. Its modular and versatile design satisfies a broad range of industry use cases. It offers a unique approach to a consensus that enables performance at scale while preserving privacy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*onM8bZmVfjSzRcxsLy09Pg.jpeg" /></figure><p>This step by step guide goes through what is needed to set up Hyperledger Fabric on an server with <em>systemd</em>. Note that, this setup does not use Docker. This is also a good learner guide on understanding how Hyperledger Fabric works from an administrators perspective.</p><h3>Environment setup</h3><p>This article explains a minimal setup with 2 hosts, but each services can live in the same/different host based on users choice. Also inorder to keep it simple, TLS is omitted in the setup, hence do not try it in production.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/db39de35bb0518535540d76036438f19/href">https://medium.com/media/db39de35bb0518535540d76036438f19/href</a></iframe><h3>Admin and Peer node(s)</h3><p>The following instructions are applicable to both the above nodes:</p><ol><li>(Optional) Add the ips of nodes to if DNS resolution is not available. In this case, I am using an example domain car.com</li></ol><pre>export ADMIN_IP=192.168.10.142<br>export PEER_IP=192.168.10.143<br>echo &quot;$ADMIN_IP orderer.car.com&quot; | sudo tee -a /etc/hosts<br>echo &quot;$PEER_IP peer0.org1.car.com&quot; | sudo tee -a /etc/hosts</pre><p>2. Install golang 1.13 and add go environment variables</p><pre>sudo add-apt-repository ppa:longsleep/golang-backports<br>sudo apt install golang-go -y<br>echo &#39;export GOPATH=$HOME/go&#39; | tee -a ~/.bashrc<br>echo &#39;export GOBIN=$GOPATH/bin&#39; | tee -a ~/.bashrc<br>source ~/.bashrc</pre><p>3. Download and install hyperledger fabric binaries (version 1.4.3) to /usr/local/bin</p><pre>mkdir ~/fabric &amp;&amp; cd ~/fabric<br>curl -sSL http://bit.ly/2ysbOFE | bash -s -- 2.0<strong>.</strong>0<strong>-</strong>beta 1.4<strong>.</strong>4 0.4<strong>.</strong>18<br>sudo cp ~/fabric/fabric-samples/bin/* /usr/local/bin/</pre><p>4. Build the fabric-ca server and install to /usr/local/bin</p><pre>go get -u github.com/hyperledger/fabric-ca/cmd/...<br>sudo cp $GOBIN/fabric-ca-server /usr/local/bin/</pre><p>5. Make the necessary destination directories for orderer, peerOrg1, users, fabric and configtx.</p><pre>sudo mkdir -p /etc/hyperledger/{configtx,fabric,config,msp}<br>sudo mkdir -p /etc/hyperledger/msp/{orderer,peerOrg1,users}</pre><p>6. Download the orderer-config archive and extract it under /etc/hyperledger/fabric. This contains MSP for orderer to start.</p><pre>curl -sSLO https://raw.githubusercontent.com/cloudronics/startFiles/master/orderer-config.tar.gz<br>sudo tar -zxvf orderer-config.tar.gz -C /etc/hyperledger/fabric/</pre><h3>Admin node</h3><h4>Generate configs genesis block etc.</h4><ol><li>Download the crypto-config and configtx yaml.</li></ol><pre>curl -sSLO <a href="https://raw.githubusercontent.com/cloudronics/startFiles/master/crypto-config.yaml">https://raw.githubusercontent.com/cloudronics/startFiles/master/crypto-config.yaml</a><br>curl -sSLO <a href="https://raw.githubusercontent.com/cloudronics/startFiles/master/configtx.yaml">https://raw.githubusercontent.com/cloudronics/startFiles/master/configtx.yaml</a></pre><p>2. Generate cryto-config, this will create necessary certificates for our car.com network</p><pre>cryptogen generate --config=./crypto-config.yaml</pre><p>3. Generate genesis block for the orderer.</p><pre>mkdir config<br>configtxgen -profile OneOrgOrdererGenesis -outputBlock ./config/genesis.block -channelID carsales-sys-channel</pre><p>4. Generate channel configuration transaction.</p><pre>configtxgen -profile OneOrgChannel -outputCreateChannelTx ./config/carsales.tx -channelID carsales</pre><p>5. Generate anchor peer transaction.</p><pre>configtxgen -asOrg Org1MSP -channelID carsales -profile OneOrgChannel -outputAnchorPeersUpdate ./config/Org1MSPanchors.tx</pre><p>6. Copy the generated configuration files to the Hyperledger directory.</p><pre>sudo cp config/* /etc/hyperledger/configtx/</pre><p>7. Also, copy the generated crypto-configs to their respective MSP directories.</p><pre>sudo cp -r crypto-config/ordererOrganizations/car.com/orderers/orderer.car.com/* /etc/hyperledger/msp/orderer/<br>sudo cp -r crypto-config/peerOrganizations/org1.car.com/peers/peer0.org1.car.com/* /etc/hyperledger/msp/peerOrg1/<br>sudo cp -r crypto-config/peerOrganizations/org1.car.com/users/* /etc/hyperledger/msp/users</pre><h4>Install Fabric-CA and Orderer service</h4><ol><li>Create a fabric-ca service by making the following service file.</li></ol><pre>cat &gt; fabric-ca.service &lt;&lt; EOF<br># Service definition for Hyperledger fabric-ca server<br>[Unit]<br>Description=hyperledger fabric-ca server - Certificate Authority for hyperledger fabric<br>Documentation=https://hyperledger-fabric-ca.readthedocs.io/<br>Wants=network-online.target<br>After=network-online.target<br>[Service]<br>Type=simple<br>Restart=on-failure<br>Environment=FABRIC_LOGGING_SPEC=INFO<br>Environment=FABRIC_CA_HOME=/etc/hyperledger/fabric-ca-server<br>Environment=FABRIC_CA_SERVER_HOME=/etc/hyperledger/fabric-ca-server<br>Environment=CA_CFG_PATH=/etc/hyperledger/fabric-ca<br>ExecStart=/usr/local/bin/fabric-ca-server start -b admin:adminpw<br>[Install]<br>WantedBy=multi-user.target<br>EOF</pre><p>2. Create a fabric orderer service by making the following service file.</p><pre>cat &gt; fabric-orderer.service &lt;&lt; EOF<br># Service definition for Hyperledger fabric orderer server<br>[Unit]<br>Description=hyperledger fabric-orderer server - Orderer for hyperledger fabric<br>Documentation=https://hyperledger-fabric.readthedocs.io/<br>Wants=network-online.target<br>After=network-online.target<br>[Service]<br>Type=simple<br>Restart=on-failure<br>Environment=FABRIC_LOGGING_SPEC=INFO<br>Environment=CA_CFG_PATH=/etc/hyperledger/fabric-ca<br>Environment=ORDERER_GENERAL_LISTENADDRESS=0.0.0.0<br>Environment=ORDERER_GENERAL_GENESISMETHOD=file<br>Environment=ORDERER_GENERAL_GENESISFILE=/etc/hyperledger/configtx/genesis.block<br>Environment=ORDERER_GENERAL_LOCALMSPID=OrdererMSP<br>Environment=ORDERER_GENERAL_LOCALMSPDIR=/etc/hyperledger/msp/orderer/msp<br>Environment=ORDERER_GENERAL_LISTENADDRESS=0.0.0.0<br>ExecStart=/usr/local/bin/orderer start<br>[Install]<br>WantedBy=multi-user.target<br>EOF</pre><p>2. Then move the service file, enable, start and verify the fabric-ca service is running.</p><pre>sudo mv fabric-ca.service /etc/systemd/system/<br>sudo systemctl enable fabric-ca.service<br>sudo systemctl start fabric-ca.service</pre><pre>sudo mv fabric-orderer.service /etc/systemd/system/<br>sudo systemctl enable fabric-orderer.service<br>sudo systemctl start fabric-orderer.service</pre><pre>systemctl status fabric-ca.service<br>systemctl status fabric-orderer.service</pre><p>3. The output of successfully running fabric services would look something like the example for fabric-ca service below.</p><pre>* fabric-ca.service - hyperledger fabric-ca server - Certificate Authority for hyperledger fabric<br>   Loaded: loaded (/etc/systemd/system/fabric-ca.service; enabled; vendor preset: enabled)<br>   Active: active (running) since Wed 2019-07-03 09:49:06 UTC; 5s ago<br>     Docs: https://hyperledger-fabric-ca.readthedocs.io/<br> Main PID: 22133 (fabric-ca-serve)<br>    Tasks: 4 (limit: 1109)<br>   CGroup: /system.slice/fabric-ca.service<br>           `-22133 /usr/local/bin/fabric-ca-server start -b admin:adminpw</pre><h3>Peer node</h3><h4>Copy crypto configs from Admin node</h4><ol><li>Set the ip address of the admin host</li></ol><pre>export ADMIN_IP=192.168.10.142</pre><p>2. Copy the config and crypto-config from the admin node to the peer.</p><pre>sudo rsync -r $USER@$ADMIN_IP:/home/ubuntu/fabric/crypto-config/peerOrganizations/org1.car.com/peers/peer0.org1.car.com/* /etc/hyperledger/msp/peerOrg1/<br>sudo rsync -r $USER@$ADMIN_IP:/home/ubuntu/fabric/crypto-config/peerOrganizations/org1.car.com/users/ /etc/hyperledger/msp/users<br>sudo rsync -r $USER@$ADMIN_IP:/etc/hyperledger/fabric/msp/ /etc/hyperledger/fabric/msp<br>sudo scp $USER@$ADMIN_IP:/home/ubuntu/fabric/fabric-samples/config/* /etc/hyperledger/configtx/<br>sudo scp $USER@$ADMIN_IP:/home/ubuntu/fabric/fabric-samples/bin/peer /usr/local/bin<br>sudo scp $USER@$ADMIN_IP:/etc/hyperledger/fabric/core.yaml /etc/hyperledger/fabric/<br>sudo scp $USER@$ADMIN_IP:/etc/hyperledger/configtx/carsales.tx /etc/hyperledger/configtx/carsales.tx</pre><p>3. (Optional) If you have to use identity file via rsync then use rsync -Pav -e &#39;ssh -i &lt;identity file&gt;&#39; -r $USER@$ADMIN_HOST:/home/.....</p><h4>Install Peer0 service</h4><p>Next, create the following service file on the peer node.</p><pre>cat &gt; fabric-peer0-org1.service &lt;&lt; EOF<br># Service definition for Hyperledger fabric peer server<br>[Unit]<br>Description=hyperledger fabric-peer0-org1 server - Peer0/Org1 for hyperledger fabric<br>Documentation=https://hyperledger-fabric.readthedocs.io/<br>Wants=network-online.target<br>After=network-online.target<br>[Service]<br>Type=simple<br>Restart=on-failure<br>Environment=FABRIC_LOGGING_SPEC=INFO<br>Environment=FABRIC_CFG_PATH=/etc/hyperledger/fabric<br>Environment=CORE_PEER_ID=peer0.org1.car.com<br>Environment=CORE_LOGGING_PEER=info<br>Environment=CORE_CHAINCODE_LOGGING_LEVEL=info<br>Environment=CORE_PEER_LOCALMSPID=Org1MSP<br>Environment=CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/msp/peerOrg1/msp<br>Environment=CORE_PEER_ADDRESS=peer0.org1.car.com:7051<br>ExecStart=/usr/local/bin/peer node start<br>[Install]<br>WantedBy=multi-user.target<br>EOF</pre><p>2. Get the service running.</p><pre>sudo mv fabric-peer0-org1.service /etc/systemd/system/<br>sudo systemctl enable fabric-peer0-org1.service<br>sudo systemctl start fabric-peer0-org1.service<br>systemctl status fabric-peer0-org1.service</pre><h4>Create a carssales channel and join peer0</h4><ol><li>The following must be done as root, switch to the root user.</li></ol><pre>sudo su -</pre><p>2. Export the necessary variables.</p><pre>echo &#39;export CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/msp/users/Admin\@org1.car.com/msp&#39; | tee -a ~/.bashrc<br>echo &#39;export CORE_PEER_LOCALMSPID=Org1MSP&#39; | tee -a ~/.bashrc<br>echo &#39;export CORE_PEER_ADDRESS=peer0.org1.car.com:7051&#39; | tee -a ~/.bashrc<br>echo &#39;export CORE_PEER_ID=peer0.org1.car.com&#39; | tee -a ~/.bashrc<br>source ~/.bashrc</pre><p>3. Verify if peer’s channels can be listed. No errors nor channels to be reported here.</p><pre>peer channel list</pre><p>4. Create the channel with a peer channel.</p><pre>peer channel create -o orderer.car.com:7050 -c carsales -f /etc/hyperledger/configtx/carsales.tx</pre><p>5. Then, join the peer0 to that channel with peer channel join command.</p><pre>peer channel join -b carsales.block</pre><p>6. Lastly, verify that the peer joined the channel successfully. You should see carsales listed now.</p><pre>peer channel list</pre><h3>Clean up</h3><ol><li>Stop services, disable and remove remove service files</li></ol><pre>sudo systemctl stop fabric-ca.service<br>sudo systemctl stop fabric-orderer.service<br>sudo systemctl stop fabric-peer0-org1.service</pre><pre>sudo systemctl disable fabric-ca.service<br>sudo systemctl disable fabric-orderer.service<br>sudo systemctl disable fabric-peer0-org1.service</pre><pre>rm -f /etc/systemd/system/fabric-ca.service<br>rm -f /etc/systemd/system/fabric-orderer.service<br>rm -f /etc/systemd/system/fabric-peer0-org1.service</pre><p>2. Remove golang</p><pre>sudo apt remove golang</pre><p>3. Remove orderer and peer entries if present in /etc/hosts</p><p>4. Remove relevant entries in ~/.bashrc</p><p>5. Remove ~/faric , /etc/hyperledger folders</p><pre>rm -rf ~/fabric<br>sudo rm -rf /etc/hyperledger</pre><p>6. Remove relevant entries from /usr/local/bin/</p><h3>Conclusion</h3><p>Now a hyperledger fabric cluster is setup with 1 org and 1 peer.</p><h3>Terms and Glossary:</h3><ol><li>Official docs: <a href="https://hyperledger-fabric.readthedocs.io/en/release-1.4/glossary.html">https://hyperledger-fabric.readthedocs.io/en/release-1.4/glossary.html</a></li><li>Simplified hacknoon’s glossary: <a href="https://hackernoon.com/hyperledger-fabric-the-20-most-important-terms-made-simple-2753f925db4">https://hackernoon.com/hyperledger-fabric-the-20-most-important-terms-made-simple-2753f925db4</a></li></ol><h3>References:</h3><ol><li>First version of the original article : <a href="https://upcloud.com/community/tutorials/hyperledger-fabric-systemd/">https://upcloud.com/community/tutorials/hyperledger-fabric-systemd/</a></li></ol><p>2. Hyperledger fabric read the docs — <a href="https://hyperledger-fabric.readthedocs.io/en/release-1.4/index.html">https://hyperledger-fabric.readthedocs.io/en/release-1.4/index.html</a></p><p>3. Hyperledger fabric certificate authority read the docs — <a href="https://hyperledger-fabric-ca.readthedocs.io/en/release-">https://hyperledger-fabric-ca.readthedocs.io/en/release-</a><br>1.4/index.html</p><p>4. File references: <a href="https://github.com/cloudronics/startFiles">https://github.com/cloudronics/startFiles</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=746359e174c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloudronics/hyperledger-fabric-blockchain-cluster-without-using-docker-containers-746359e174c">Hyperledger fabric blockchain cluster without using docker containers</a> was originally published in <a href="https://medium.com/cloudronics">Cloudronics</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to create elastic beat from your code snippet — part 2]]></title>
            <link>https://medium.com/cloudronics/how-to-create-elastic-beat-from-your-code-snippet-part-2-b81d965079ae?source=rss----63dd8998b468---4</link>
            <guid isPermaLink="false">https://medium.com/p/b81d965079ae</guid>
            <category><![CDATA[filebeat]]></category>
            <category><![CDATA[elastic]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[kibana]]></category>
            <category><![CDATA[beats]]></category>
            <dc:creator><![CDATA[anoop vijayan maniankara]]></dc:creator>
            <pubDate>Wed, 12 Jun 2019 08:26:56 GMT</pubDate>
            <atom:updated>2019-06-12T08:26:56.252Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to create elastic beat from your code snippet — part 2</h3><p>This is continuation from the <a href="https://medium.com/cloudronics/convert-your-code-snippet-to-an-elastic-beat-in-8-simple-steps-54dfa2beb67c">previous post</a>.</p><h3>Table of Contents</h3><ul><li><a href="https://medium.com/p/b81d965079ae#91e4">Recap</a></li><li><a href="https://medium.com/p/b81d965079ae#0067">Versions</a></li><li><a href="https://medium.com/p/b81d965079ae#16c7">Launching elastic stack locally</a></li><li><a href="https://medium.com/p/b81d965079ae#4caa">Launching fileoccurencebeat locally</a></li><li><a href="https://medium.com/p/b81d965079ae#94d5">Conclusions</a></li><li><a href="https://medium.com/p/b81d965079ae#4d4d">References</a></li></ul><h3>Recap</h3><p>In the <a href="https://medium.com/cloudronics/convert-your-code-snippet-to-an-elastic-beat-in-8-simple-steps-54dfa2beb67c">previous post</a>, we managed to implement our own <em>fileoccurancebeat</em>. Now its time for some testing.</p><h3>Versions</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7b8c60c0ede4d6dec791bc4e7f8eed83/href">https://medium.com/media/7b8c60c0ede4d6dec791bc4e7f8eed83/href</a></iframe><h3>Launching elastic stack locally</h3><p>We will use docker containers for this. Also, for this test, we just need Elasticsearch and Kibana, so EK stack is enough. Lets launch them:</p><pre>curl <a href="https://raw.githubusercontent.com/cloudronics/fileoccurancebeat/master/elasticstack-docker.yaml">https://raw.githubusercontent.com/cloudronics/fileoccurancebeat/master/elasticstack-docker.yaml</a> &gt;docker-compose.yaml &amp;&amp; docker-compose up -d</pre><p>This will get you running elasticsearch 7.1 and Kibana 7.1 running on ports 9200 and 5601 respectively.</p><h4>Verify Elasticsearch</h4><p>Lets verify if elasticsearch launched successfully and running.</p><pre>$ curl localhost:9200/_cluster/health?pretty<br>{<br>  &quot;cluster_name&quot; : &quot;docker-cluster&quot;,<br>  &quot;status&quot; : &quot;green&quot;,<br>  &quot;timed_out&quot; : false,<br>  &quot;number_of_nodes&quot; : 1,<br>  &quot;number_of_data_nodes&quot; : 1,<br>  &quot;active_primary_shards&quot; : 1,<br>  &quot;active_shards&quot; : 1,<br>  &quot;relocating_shards&quot; : 0,<br>  &quot;initializing_shards&quot; : 0,<br>  &quot;unassigned_shards&quot; : 0,<br>  &quot;delayed_unassigned_shards&quot; : 0,<br>  &quot;number_of_pending_tasks&quot; : 0,<br>  &quot;number_of_in_flight_fetch&quot; : 0,<br>  &quot;task_max_waiting_in_queue_millis&quot; : 0,<br>  &quot;active_shards_percent_as_number&quot; : 100.0<br>}</pre><h4>Verify Kibana</h4><p>Lets try to do the same for Kibana too.</p><pre>$ curl -i localhost:5601/<br>HTTP/1.1 302 Found<br>location: /app/kibana<br>kbn-name: kibana<br>kbn-xpack-sig: f06b19db5de93fafdcbc1d7c1ef4aa3e<br>content-type: text/html; charset=utf-8<br>cache-control: no-cache<br>content-length: 0<br>connection: close<br>Date: Sun, 02 Jun 2019 06:25:27 GMT</pre><h4>Verify Kibana from browser</h4><p>Now, open <a href="http://localhost:5601/">http://localhost:5601/</a> in browser.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TGMtrxR5KZnF88T27Ae31Q.png" /></figure><h3>Launching fileoccurencebeat locally</h3><p>Now that we have our local E(L)K stack running, lets lauch our fileoccurencebeat from the <a href="https://medium.com/cloudronics/convert-your-code-snippet-to-an-elastic-beat-in-8-simple-steps-54dfa2beb67c">previous post</a>. As we are running on localhost and are using default ports, we do not have to configure anything.</p><h4>Adding tmp root path</h4><p>Lets add a custom <em>rootpath</em> and <em>filename</em> to the beat in <em>fileoccurencebeat.yml</em>. Increase the period to <em>20 seconds, </em>add <em>rootpath</em> and <em>filename. R</em>eplace:</p><pre>fileoccurencebeat:</pre><pre><em>  # Defines how often an event is sent to the output</em></pre><pre>  period: 1s</pre><p>with</p><pre>fileoccurencebeat:</pre><pre><em>  # Defines how often an event is sent to the output</em></pre><pre>  period: 20s</pre><pre>  rootpath: /tmp/</pre><pre>  filename: abc</pre><h4>Start the fileoccurencebeat</h4><pre>$ ./fileoccurencebeat -e -d &#39;*&#39;<br>2019-06-02T11:37:49.574+0300 INFO instance/beat.go:606 Home path: [/home/go/src/github.com/cloudronics/fileoccurencebeat] Config path: [/home/go/src/github.com/cloudronics/fileoccurencebeat] Data path: [/home/go/src/github.com/cloudronics/fileoccurencebeat/data] Logs path: [/home/go/src/github.com/cloudronics/fileoccurencebeat/logs]<br>2019-06-02T11:37:49.575+0300 INFO instance/beat.go:614 Beat ID: ef1f96bf-1d6f-4059-a40c-0a76be79acfb<br>2019-06-02T11:37:49.577+0300 INFO [beat] instance/beat.go:902 Beat info {&quot;system_info&quot;: {&quot;beat&quot;: {&quot;path&quot;: {&quot;config&quot;: &quot;/home/go/src/github.com/cloudronics/fileoccurencebeat&quot;, &quot;data&quot;: &quot;/home/go/src/github.com/cloudronics/fileoccurencebeat/data&quot;, &quot;home&quot;: &quot;/home/go/src/github.com/cloudronics/fileoccurencebeat&quot;, &quot;logs&quot;: &quot;/home/go/src/github.com/cloudronics/fileoccurencebeat/logs&quot;}, &quot;type&quot;: &quot;fileoccurencebeat&quot;, &quot;uuid&quot;: &quot;ef1f96bf-1d6f-4059-a40c-0a76be79acfb&quot;}}}</pre><p>Leave it running on that terminal and use another terminal for testing.</p><h4>Configure Kibana to receive beats</h4><p>In Kibana, <a href="https://www.elastic.co/guide/en/kibana/current/tutorial-define-index.html">create index pattern</a> of format <em>fileoccurencebeat-8.0.0–yyyy.mm.dd–N</em>. In my case, it was <em>fileoccurencebeat-8.0.0–2019.06.02–000001</em>. Select \<em>@timestamp</em> as the time filter field name. Open the discover tab to see the events arriving.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PASj3EZFRlsgBbgyVg9LKA.png" /></figure><h4>Create Kibana visuals for file occurences</h4><p>If you are familiar with Kibana charts, this should be simpler.</p><ol><li>Open Visualize -&gt; Create a visualization -&gt; Line charts -&gt; Select Index (filebeatoccurence-). You will end up with an empty chart with Count.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-6uPNFLPt4M8bwEUk5Xpeg.png" /></figure><p>2. Configure X-axis to <em>Date Histogram</em> and Y-Axis to <em>Average</em> of <em>occurrences</em>. Set the refresh rate to 1 second and set the time from 2/5 minutes — now. From below you can see that there is no occurrences of file <em>abc</em> under /<em>tmp</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f-DayltgDcqWIVEYblBHQQ.png" /></figure><h4>Perform a simple test</h4><p>Try adding files with name abc under /tmp and observe what happens</p><pre>$ touch /tmp/abc<br>$ mkdir /tmp/xyz<br>$ touch /tmp/xyz/abc<br>$ mkdir /mno<br>$ touch /tmp/mno/abc</pre><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FUPaJdBv2LjQ%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DUPaJdBv2LjQ&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FUPaJdBv2LjQ%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/8f524ecb93881fe349835fd8efc076ec/href">https://medium.com/media/8f524ecb93881fe349835fd8efc076ec/href</a></iframe><p>You can see from the above video that, upon creating a files with name <em>abc</em> under /tmp/, you see the graph starts to show the count of occurences in few seconds.</p><h3>Conclusions and further reading</h3><p>Now we managed to successfully test the beats generated in our local environment. Go ahead and publish your beat to <a href="https://www.elastic.co/guide/en/beats/libbeat/7.1/community-beats.html">elastic community beats</a>.</p><h3>References</h3><ol><li>Elastic community beats — <a href="https://www.elastic.co/guide/en/beats/libbeat/7.1/community-beats.html">https://www.elastic.co/guide/en/beats/libbeat/7.1/community-beats.html</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b81d965079ae" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloudronics/how-to-create-elastic-beat-from-your-code-snippet-part-2-b81d965079ae">How to create elastic beat from your code snippet — part 2</a> was originally published in <a href="https://medium.com/cloudronics">Cloudronics</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Convert your code snippet to an elastic beat in 8 simple steps]]></title>
            <link>https://medium.com/cloudronics/convert-your-code-snippet-to-an-elastic-beat-in-8-simple-steps-54dfa2beb67c?source=rss----63dd8998b468---4</link>
            <guid isPermaLink="false">https://medium.com/p/54dfa2beb67c</guid>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[beats]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[elastic]]></category>
            <dc:creator><![CDATA[anoop vijayan maniankara]]></dc:creator>
            <pubDate>Fri, 31 May 2019 13:05:46 GMT</pubDate>
            <atom:updated>2019-05-31T13:05:46.823Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to create elastic beat from your code snippet</h3><p><em>This article is based on the </em><a href="https://www.elastic.co/guide/en/beats/devguide/7.2/index.html"><em>beats developer guide</em></a><em>. Elastic supports only golang for beats at the time of writing. The full source code for this beat is available in </em><a href="https://github.com/cloudronics/fileoccurancebeat"><em>cloudronics github repository</em></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QH6K7c8t1YcGjnH3BMN3xQ.jpeg" /><figcaption>elastic.co</figcaption></figure><p>Elastic beats are shippers which sits on servers, containers etc. gathering data from the host and sending it to elasticsearch server. Read more about it <a href="https://www.elastic.co/products/beats">here</a>.</p><h4>Table of Contents</h4><ul><li><a href="https://medium.com/p/54dfa2beb67c#1903">Versions</a></li><li><a href="https://medium.com/p/54dfa2beb67c#896a">Introduction</a></li><li><a href="https://medium.com/p/54dfa2beb67c#32ca">Check for file availability on a server — snippet</a></li><li><a href="https://medium.com/p/54dfa2beb67c#9f4c">Convert the snippet to Elastic beat</a></li><li><a href="https://medium.com/p/54dfa2beb67c#e600">Conclusion and further reading</a></li><li><a href="https://medium.com/p/54dfa2beb67c#8e88">References</a></li></ul><h4>Versions</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7b8c60c0ede4d6dec791bc4e7f8eed83/href">https://medium.com/media/7b8c60c0ede4d6dec791bc4e7f8eed83/href</a></iframe><h3>Introduction</h3><p>Lets say you have some code which analyses your system and you would like to monitor the same in a time series fashion i.e. the outcome over a limited/unlimited duration. Here I am going to give an example of <em>check for file availability on a server</em>. This snippet notifies when a file appears on a server. This case is useful for monitoring queues, buffers etc.</p><h3>Check for file availability on a server — snippet</h3><p>Here is a example snippet which finds and prints the occurrence of a fileabc under /home/go directory.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/07683c6398db6c293ae9fb7bdd544833/href">https://medium.com/media/07683c6398db6c293ae9fb7bdd544833/href</a></iframe><h3>Convert the snippet to Elastic beat</h3><p>Lets first name the beat: <strong>fileoccurancebeat</strong>.</p><ol><li>Export env variables</li></ol><pre>export USER=user1<br>export GOPATH=/home/${USER}/go</pre><p>2. Create directories and clone the elastic beat repository</p><pre>mkdir -p ${GOPATH}/src/github.com/elastic<br>git clone https://github.com/elastic/beats ${GOPATH}/src/github.com/elastic/beats<br>git checkout 7.2</pre><p>3. Generate the beat</p><pre>mkdir ${GOPATH}/src/github.com/${USER}<br>cd ${GOPATH}/src/github.com/${USER}<br>python $GOPATH/src/github.com/elastic/beats/script/generate.py<br>Beat Name [Examplebeat]: Fileoccurancebeat<br>Your Github Name [your-github-name]: cloudronics<br>Beat Path [github.com/cloudronics/fileoccurancebeat]:<br>Firstname Lastname: Cloudronics Oy</pre><p>4. Set it up</p><pre>cd ${GOPATH}/src/github.com/cloudronics/fileoccurancebeat<br>make setup</pre><p>5. Now we have a boilerplate setup to start with. The following files needed to be modified:</p><ul><li><em>_meta/beat.yaml</em> — The template file from which the configuration is generated</li><li><em>config/config.go </em>— Struct defining the above config</li><li><em>beater/fileoccurancebeat.go</em> — Actual implementation resides here</li></ul><p>6. Include our snippet as a function fileOccurences and add it to the event</p><pre>// Checks for a given file under a given path recurssively</pre><pre>func fileOccurences(rootPath string, fileName string) int64 {</pre><pre>  var count int64 = 0</pre><pre>  err := filepath.Walk(rootPath, func(path string, info os.FileInfo, err error) error {</pre><pre>    if err != nil {</pre><pre>      logp.Error(err)</pre><pre>      return err</pre><pre>    }</pre><pre>    if info.Name() == fileName {</pre><pre>      count++</pre><pre>    }</pre><pre>    return nil</pre><pre>  })</pre><pre>  if err != nil {</pre><pre>    logp.Error(err)</pre><pre>  return 0</pre><pre>  }</pre><pre>  return count</pre><pre>}</pre><p>7. An option to provide the root directory and the file name from config needs to be included. Add them to config.go and beat.yaml template and run the build with mage (yes not make):</p><pre>mage build</pre><p>8. Now try to launch fileoccurencebeat locally (Ctl+c to stop):</p><pre>./fileoccurencebeat -e -d &quot;*&quot;</pre><h3>Conclusion and further reading</h3><p>Now we have a working beat, in the next article we can test the beat metrics, plot graphs etc. with an elastic stack running locally.</p><h3>References</h3><ol><li>Beats developer guide from Elastic — <a href="https://www.elastic.co/guide/en/beats/devguide/7.2/index.html">https://www.elastic.co/guide/en/beats/devguide/7.2/index.html</a></li><li>What are elastic beats — <a href="https://www.elastic.co/products/beats">https://www.elastic.co/products/beats</a></li><li>Github repository — <a href="https://github.com/cloudronics/fileoccurancebeat">https://github.com/cloudronics/fileoccurancebeat</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=54dfa2beb67c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloudronics/convert-your-code-snippet-to-an-elastic-beat-in-8-simple-steps-54dfa2beb67c">Convert your code snippet to an elastic beat in 8 simple steps</a> was originally published in <a href="https://medium.com/cloudronics">Cloudronics</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>