The Log: What Every Software Engineer Should Know About Real-Time Data's Unifying Abstraction
- Authors
-
-
Alkis Samaras
-
- Abstract
-
Applications increasingly need to process data as continuous streams rather than as finite datasets. This paper argues that an append-only log provides a fundamental abstraction for building distributed systems. By using logs as the central data structure, systems can simplify replication, fault tolerance, stream processing, data integration, and event sourcing. The paper discusses how technologies such as Apache Kafka leverage the log abstraction to build scalable and reliable backend infrastructures for modern applications.
- References
-
DeCandia, G., Hastorun, D., Jampani, M., Kakulapati, G., Lakshman, A., Pilchin, A., et al. (2007). Dynamo: Amazon's highly available key-value store. Proceedings of the 21st ACM SIGOPS Symposium on Operating Systems Principles (SOSP '07), 205–220. https://doi.org/10.1145/1294261.1294281
Chang, F., Dean, J., Ghemawat, S., Hsieh, W. C., Wallach, D. A., Burrows, M., et al. (2008). Bigtable: A distributed storage system for structured data. ACM Transactions on Computer Systems, 26(2), Article 4. https://doi.org/10.1145/1365815.1365816
Ghemawat, S., Gobioff, H., & Leung, S.-T. (2003). The Google File System. Proceedings of the 19th ACM Symposium on Operating Systems Principles (SOSP '03), 29–43. https://doi.org/10.1145/945445.945450
Dean, J., & Ghemawat, S. (2004). MapReduce: Simplified data processing on large clusters. Proceedings of the 6th Symposium on Operating Systems Design and Implementation (OSDI), 137–150.
Lakshman, A., & Malik, P. (2010). Cassandra: A decentralized structured storage system. ACM SIGOPS Operating Systems Review, 44(2), 35–40. https://doi.org/10.1145/1773912.1773922
Corbett, J. C., Dean, J., Epstein, M., Fikes, A., Frost, C., Furman, J. J., et al. (2013). Spanner: Google's globally distributed database. ACM Transactions on Computer Systems, 31(3), Article 8. https://doi.org/10.1145/2491245
Ongaro, D., & Ousterhout, J. (2014). In Search of an Understandable Consensus Algorithm (Raft). Proceedings of the USENIX Annual Technical Conference (ATC), 305–319.
Gilbert, S., & Lynch, N. (2002). Brewer's conjecture and the feasibility of consistent, available, partition-tolerant web services. ACM SIGACT News, 33(2), 51–59. https://doi.org/10.1145/564585.564601
Vogels, W. (2009). Eventually Consistent. Communications of the ACM, 52(1), 40–44. https://doi.org/10.1145/1435417.1435432
Kreps, J., Narkhede, N., & Rao, J. (2011). Kafka: A distributed messaging system for log processing. Proceedings of the NetDB Workshop, 1–7.
Barlocker, B., & Yan, X. (2021). Contrastive Representation Learning for Anomaly Detection in Cloud-Based Backend Services. Artificial Intelligence and Computing Innovations, 1(2).
- Published
- 2022-04-30
- Issue
- Vol. 1 No. 2 (2022)
- Section
- Articles
- License
-

This work is licensed under a Creative Commons Attribution 4.0 International License.
Academic Scholar Hub Press also operates under the Creative Commons Licence CC-BY 4.0. This allows for copy and redistribute the material in any medium or format for any purpose, even commercially. The premise is that you must provide appropriate citation information.
