The Log: What Every Software Engineer Should Know About Real-Time Data's Unifying Abstraction

Authors
  • Alkis Samaras

Abstract

Applications increasingly need to process data as continuous streams rather than as finite datasets. This paper argues that an append-only log provides a fundamental abstraction for building distributed systems. By using logs as the central data structure, systems can simplify replication, fault tolerance, stream processing, data integration, and event sourcing. The paper discusses how technologies such as Apache Kafka leverage the log abstraction to build scalable and reliable backend infrastructures for modern applications.

References

DeCandia, G., Hastorun, D., Jampani, M., Kakulapati, G., Lakshman, A., Pilchin, A., et al. (2007). Dynamo: Amazon's highly available key-value store. Proceedings of the 21st ACM SIGOPS Symposium on Operating Systems Principles (SOSP '07), 205–220. https://doi.org/10.1145/1294261.1294281

Chang, F., Dean, J., Ghemawat, S., Hsieh, W. C., Wallach, D. A., Burrows, M., et al. (2008). Bigtable: A distributed storage system for structured data. ACM Transactions on Computer Systems, 26(2), Article 4. https://doi.org/10.1145/1365815.1365816

Ghemawat, S., Gobioff, H., & Leung, S.-T. (2003). The Google File System. Proceedings of the 19th ACM Symposium on Operating Systems Principles (SOSP '03), 29–43. https://doi.org/10.1145/945445.945450

Dean, J., & Ghemawat, S. (2004). MapReduce: Simplified data processing on large clusters. Proceedings of the 6th Symposium on Operating Systems Design and Implementation (OSDI), 137–150.

Lakshman, A., & Malik, P. (2010). Cassandra: A decentralized structured storage system. ACM SIGOPS Operating Systems Review, 44(2), 35–40. https://doi.org/10.1145/1773912.1773922

Corbett, J. C., Dean, J., Epstein, M., Fikes, A., Frost, C., Furman, J. J., et al. (2013). Spanner: Google's globally distributed database. ACM Transactions on Computer Systems, 31(3), Article 8. https://doi.org/10.1145/2491245

Ongaro, D., & Ousterhout, J. (2014). In Search of an Understandable Consensus Algorithm (Raft). Proceedings of the USENIX Annual Technical Conference (ATC), 305–319.

Gilbert, S., & Lynch, N. (2002). Brewer's conjecture and the feasibility of consistent, available, partition-tolerant web services. ACM SIGACT News, 33(2), 51–59. https://doi.org/10.1145/564585.564601

Vogels, W. (2009). Eventually Consistent. Communications of the ACM, 52(1), 40–44. https://doi.org/10.1145/1435417.1435432

Kreps, J., Narkhede, N., & Rao, J. (2011). Kafka: A distributed messaging system for log processing. Proceedings of the NetDB Workshop, 1–7.

Barlocker, B., & Yan, X. (2021). Contrastive Representation Learning for Anomaly Detection in Cloud-Based Backend Services. Artificial Intelligence and Computing Innovations, 1(2).

Cover Image
Published
2022-04-30
Section
Articles
License
Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

Academic Scholar Hub Press also operates under the Creative Commons Licence CC-BY 4.0. This allows for copy and redistribute the material in any medium or format for any purpose, even commercially. The premise is that you must provide appropriate citation information.

How to Cite

Samaras, A. (2022). The Log: What Every Software Engineer Should Know About Real-Time Data’s Unifying Abstraction. Journal of Computer Technology and Software, 1(2). https://ashpress.net/index.php/jcts/article/view/337