{"id":527,"date":"2018-09-10T10:20:26","date_gmt":"2018-09-10T01:20:26","guid":{"rendered":"https:\/\/dong1lkim.oboki.net\/?p=527"},"modified":"2021-01-25T22:22:23","modified_gmt":"2021-01-25T13:22:23","slug":"kafka-apache-kafka","status":"publish","type":"post","link":"https:\/\/oboki.net\/workspace\/data-engineering\/kafka\/kafka-apache-kafka\/","title":{"rendered":"[Kafka] Apache Kafka"},"content":{"rendered":"<h1>Apache Kafka<\/h1>\n<h2>Introduction<\/h2>\n<h3>\uc774\ubc88\uc5d0 \uc9c1\uc811 \uc0ac\uc6a9\ud574\ubcf4\ub2c8..<\/h3>\n<p>Message Queue Open Source Framework\ub85c\uc11c kafka\ub97c \ub9ce\uc774\ub4e4 \uc0ac\uc6a9\ud558\ub294 \uac83 \uac19\ub2e4.<br \/>\n\uae30\ubcf8\uc801\uc73c\ub85c Producer\/Consumer Application \uc744 \uac1c\ubc1c\ud574\uc11c \uc0ac\uc6a9\ud574\uc57c \ud558\ub294\ub370 \ub9ce\uc774 \uc0ac\uc6a9\ud558\ub2e4 \ubcf4\ub2c8 \uad00\ub828 \uc790\ub8cc\uac00 \ub9ce\uace0 Hadoop, Spark, \uc5ec\ub7ec \uc218\uc9d1\uae30 \ub4f1\uc5d0\uc11c Plugin\uc744 \uc81c\uacf5\ud55c\ub2e4.<br \/>\n\ub9ce\uc774 \uc4f0\uace0 \uc790\ub8cc\uac00 \ub9ce\uc544\uc11c kafka\ub97c \uc774\uc6a9\ud558\ub294 \uac83\ub3c4 \uc88b\uc740 \uc774\uc720\uc778 \uac83 \uac19\ub2e4.<\/p>\n<p>\ub2e4\uc74c \ub450\uac00\uc9c0 \uc774\uc720\uac00 \uc9c1\uc811 \ub290\ub080 \uc88b\uc740 \uc810<\/p>\n<ol>\n<li>Consumer\ub294 \ud544\uc694\ud55c \uba54\uc2dc\uc9c0\ub97c PULL \ud558\ub294 \uad6c\uc870<\/li>\n<li>\ubb34\uc911\ub2e8 \uc11c\ube44\uc2a4 \uad6c\ud604 \uac00\ub2a5<\/li>\n<\/ol>\n<h4>Consumer\ub294 \ud544\uc694\ud55c \uba54\uc2dc\uc9c0\ub97c PULL \ud558\ub294 \uad6c\uc870<\/h4>\n<p>Producer\ub294 Message\ub97c Kafka Topic\uc5d0 PUSH \ud558\uace0,<br \/>\nConsumer\ub294 Message\ub97c Kafka Topic\uc5d0\uc11c <strong>PULL<\/strong> \ud558\ub294 \uad6c\uc870\uc774\ub2e4.<\/p>\n<p>\uc774 \ub54c\ubb38\uc5d0 Consumer Application \uc774 \ud544\uc694\ub85c \ud560 \ub54c\uc5d0 \ud544\uc694\ud55c \ub9cc\ud07c\uc758 \ub370\uc774\ud130\ub97c \uac00\uc838\ub2e4\uac00 \uc4f8 \uc218 \uc788\ub2e4.<\/p>\n<h4>\ubb34\uc911\ub2e8 \uc11c\ube44\uc2a4 \uad6c\ud604 \uac00\ub2a5<\/h4>\n<p>Fluentd,Logstash \ub4f1\uc5d0\uc11c\ub3c4 Persistent Queue \uae30\ub2a5\uc744 \uac00\uc9c0\uace0\ub294 \uc788\uc9c0\ub9cc, member node\uac00 \ub2e4\uc6b4\ub418\uc5c8\uc744 \ub54c\uc5d0 \ud574\ub2f9 \ub178\ub4dc\uc5d0\uc11c \uc7a0\uc2dc\ub77c\ub3c4 \ubc84\ud37c\ub9c1 \ud55c \ub370\uc774\ud130\ub294 \uadf8 \ub178\ub4dc\uac00 \uc0b4\uc544\ub098\uc57c\ub9cc \ub2e4\uc2dc \uc804\uc1a1\ud560 \uc218 \uc788\ub2e4.<br \/>\n\uc218\uc9d1\uae30\ub9cc \uc0ac\uc6a9\ud588\uc744 \ub54c\uc5d0\ub294 \ub370\uc774\ud130\uac00 \uc720\uc2e4\ub418\uc9c0\ub294 \uc54a\uc9c0\ub9cc \uc2a4\ud2b8\ub9bc \ub370\uc774\ud130\uc758 \ud750\ub984\uc774 \uae68\uc9d0\uc73c\ub85c\uc368 \uc11c\ube44\uc2a4 \uc911\ub2e8\uc774 \uc788\ub2e4.<\/p>\n<h3>@Kakao<\/h3>\n<p>\uce74\uce74\uc624\uc5d0\uc11c\ub294 \uc544\ub798\uc640 \uac19\uc774 \ub9ce\uc740 \uba54\uc2dc\uc9c0\ub97c \uce74\ud504\uce74\ub97c \uacbd\uc720\uc2dc\ucf1c \uc11c\ube44\uc2a4\uc5d0 \ud65c\uc6a9\ud558\uace0 \uc788\ub2e4\uace0 \ud55c\ub2e4.<\/p>\n<p><img decoding=\"async\" src=\".\/images\/kakao_240_370.png\" alt=\"270_370\" \/><\/p>\n<ul>\n<li>7 Cluster \/ 130 Node<br \/>\n\uc804\uc0ac\uc801\uc73c\ub85c \ub2e4\uc591\ud55c \uc11c\ube44\uc2a4\uc5d0 \ud65c\uc6a9<\/li>\n<li>260,000,000,000 (messages \/ day)\n<ul>\n<li>in: 240 TB\/day<\/li>\n<li>out: 370 TB\/day<\/li>\n<\/ul>\n<\/li>\n<li>99.99% \uac00\ub3d9\ub960<\/li>\n<\/ul>\n<h2>Core APIs<\/h2>\n<p>Kafka has four core APIs<\/p>\n<p><img decoding=\"async\" src=\"http:\/\/kafka.apache.org\/11\/images\/kafka-apis.png\" alt=\"kafka_apis\" \/><\/p>\n<p>0.10.0 \ubc84\uc804\ubd80\ud130 Stream API, Connector API\uac00 \ucd94\uac00\ub418\uc5c8\ub2e4. Producer,Consumer API \uac00 \ud575\uc2ec \uad6c\uc870\uc774\uba70, Producer\/Consumer API\ub97c \ud65c\uc6a9\ud574 Producer\/Consumer Application \uc744 \uac1c\ubc1c\ud558\ub294 \uac83\uc774 \uae30\ubcf8.<\/p>\n<p>\ub300\ubd80\ubd84\uc758 \uc218\uc9d1\uae30\uc5d0\uc11c kafka in\/out plugin \uc744 \uc81c\uacf5\ud558\uae30 \ub54c\ubb38\uc5d0 \uac04\ud3b8\ud558\uac8c \uc0ac\uc6a9\ud558\uae30 \uc704\ud574\uc11c\ub294 \uc774\ub4e4\uc744 \uc774\uc6a9\ud558\uba74 \ud3b8\ud558\ub2e4.<\/p>\n<p>Kafka Cluster \ub294 \uc218\ub3d9\uc801\uc774\ub2e4. \uc5b4\ub5bb\uac8c \ubcf4\uba74 DBMS\uc640 \ube44\uc2b7\ud55c\ub370<\/p>\n<blockquote><p>\n  \ub370\uc774\ud130\ub97c \uc8fc\uba74 \ubc1b\uace0 \uac00\uc838\uac00\uaca0\ub2e4\uace0 \ud558\uba74 \ub0b4\uc5b4\uc900\ub2e4.\n<\/p><\/blockquote>\n<ul>\n<li>Producer \uc5b4\ud50c\ub9ac\ucf00\uc774\uc158\uc740 \uc2a4\ud2b8\ub9bc \ub370\uc774\ud130\ub97c &#8216;push&#8217; \ud558\ub294 \ud615\ud0dc\ub85c \ub3d9\uc791.  <\/li>\n<li>Consumer \uc5b4\ud50c\ub9ac\ucf00\uc774\uc158\uc740 kafka topic\uc5d0 \uc800\uc7a5\ub41c \ub370\uc774\ud130\ub97c consumer\uac00 \uc6d0\ud558\ub294 \uc8fc\uae30\/offset \ub4f1\uc744 \uc124\uc815\ud558\uc5ec \uac00\uc838\uac00\ub294(<strong>PULL<\/strong>) \ud615\ud0dc\ub85c \ub3d9\uc791.<\/li>\n<\/ul>\n<h3>Producer API<\/h3>\n<p>kafka topic\uc5d0 \uc2a4\ud2b8\ub9bc \ub370\uc774\ud130\ub97c publish\ud558\ub294 API.<\/p>\n<ul>\n<li>\ud2b9\uc815 Topic \uc758 Leader\uc5d0\uac8c \uba54\uc2dc\uc9c0 \uc804\uc1a1 (<strong>PUSH<\/strong>)\n<ul>\n<li>ACKS<\/li>\n<li>0<br \/>\n\ubcf4\ub0b4\uae30\ub9cc \ud568<\/li>\n<li>1 (default, freq)<br \/>\n\ud30c\ud2f0\uc158\uc758 \ub9ac\ub354\uac00 \ub370\uc774\ud130\ub97c \uc218\uc2e0\ud588\ub294\uc9c0\ub9cc \ud655\uc778<\/li>\n<li>ALL<br \/>\n\ubaa8\ub4e0 replication\uc774 \ub370\uc774\ud130\ub97c \uc218\uc2e0\ud588\ub294\uc9c0 \ud655\uc778<\/li>\n<\/ul>\n<\/li>\n<li>\ud2b9\uc815 \ud30c\ud2f0\uc158 \ub610\ub294 \ub79c\ub364 \ud30c\ud2f0\uc158\uc73c\ub85c \uc804\uc1a1<\/li>\n<li>\ube60\ub978 \uc804\uc1a1 \uc18d\ub3c4 \ubcf4\uc7a5<\/li>\n<li>\ud6a8\uc728\uc131\uc774 \uc88b\uc740 \ubc30\uce58 \ucc98\ub9ac \uac00\ub2a5<\/li>\n<li>\uc124\uc815\uc744 \ud1b5\ud574 \ubc30\uce58 \ud06c\uae30\ub098 \uc9c0\uc5f0\uc2dc\uac04 \uc870\uc815<\/li>\n<\/ul>\n<h3>Consumer API<\/h3>\n<p>kafka topic\uc744 subscribe\ud558\ub294 API.<\/p>\n<ul>\n<li>\ud2b9\uc815 Topic \uc758 Leader\uc5d0\uac8c <code>Fetch<\/code> \ud574\ub2ec\ub77c\uace0 \uc694\uccad (<strong>PULL<\/strong>)<\/li>\n<li>\ud574\ub2f9 topic\uc758 \ud2b9\uc815 offset\ubd80\ud130 \uc694\uccad<\/li>\n<\/ul>\n<p>\uc5b4\ub514\uc11c\ubd80\ud130 \uad6c\ub3c5\ud560 \uac83\uc778\uc9c0\ub294 Consumer Application \uc5d0\uc11c \uacb0\uc815\ud558\ub294\ub370, \uacfc\uac70\uc758 offset\uc73c\ub85c reset \uc2dc\ucf1c\uc11c \ub2e4\uc2dc \uac00\uc838\uc62c \uc9c0, \ucd5c\uadfc\uc758 \ub370\uc774\ud130\ubd80\ud130\ub9cc \uac00\uc838\uc62c \uac83\uc778\uc9c0\ub97c \uc815\ud560 \uc218 \uc788\ub2e4.<\/p>\n<p>~~spark streaming\uc758 KafkaUtils\uc5d0\uc11c offset \uc744 \ucee8\ud2b8\ub864 \ud558\ub294 \ubc29\ubc95 \uc54c\uc544\ubd10\uc57c\uaca0\ub2e4. consumer\uac00 \uc8fd\uc5c7\uc744 \ub54c\uc5d0 \ub2e4\uc2dc app\uc744 \uc0b4\ub9ac\ub354\ub77c\ub3c4 \uadf8\ub54c \uae30\uc900 \ucd5c\uc2e0 \uc790\ub8cc\ubd80\ud130 \uac00\uc838\uc62c \ub4ef.~~<br \/>\n~~\uc8fd\uc740 \uc2dc\uc810\uc758 \ub370\uc774\ud130\ubd80\ud130 \ub2e4\uc2dc \ucc98\ub9ac \ud558\ub294 \ubc29\ubc95\uc744 \uc54c\uc544\ub0b4\uc57c \ud560 \ub4ef.~~<\/p>\n<h4>Consumer Group<\/h4>\n<p>![](http:\/\/kafka.apache.org\/11\/images\/consumer-groups.png#mdimgcenter)<\/p>\n<ul>\n<li>\ud558\ub098\uc758 \ud1a0\ud53d\uc744 group \ub0b4 \uc5ec\ub7ec Consumer \ub4e4\uc774 \ub098\ub204\uc5b4 \uad6c\ub3c5<br \/>\nProducer\uac00 Topic\uc73c\ub85c \ubcf4\ub0b4\ub294 \uc18d\ub3c4\uac00 \ube60\ub978 \uacbd\uc6b0 Consumer \ud655\uc7a5 \uac00\ub2a5<\/li>\n<li>\ud558\ub098\uc758 \ud30c\ud2f0\uc158\uc5d0\ub294 \ud558\ub098\uc758 \ucee8\uc288\uba38\ub9cc \uac00\ub2a5<br \/>\n\ub3d9\uc77c\ud55c \ub370\uc774\ud130\ub97c \ucee8\uc288\uba38 \uadf8\ub8f9\uc774 \uc911\ubcf5\ud574\uc11c \uad6c\ub3c5\ud558\uc9c0 \uc54a\ub294\ub2e4.<\/li>\n<\/ul>\n<h3>Stream API<\/h3>\n<p>kafka \ud074\ub7ec\uc2a4\ud130 \uc790\uccb4\uc801\uc73c\ub85c stream \ub370\uc774\ud130 \ucc98\ub9ac \ud504\ub85c\uc138\uc2a4\ub97c \uac16\ub294\ub2e4. \uc544\ub798\uc640 \uac19\uc740 \uc7a5\uc810\uc774 \uc788\ub2e4\uace0\ub4e4 \ud55c\ub2e4.<\/p>\n<ul>\n<li>Very light weight library, good for microservices, IOT applications<\/li>\n<li>Exactly Once<\/li>\n<li>Does not need dedicated cluster<\/li>\n<li>Inherits all Kafka Good characteristics<\/li>\n<li>Supports Stream joins, internally uses rocksDB for maintaining state.<\/p>\n<\/li>\n<li>\n<p>tightly coupled with Kafka, can not use without Kafka in picture<\/p>\n<\/li>\n<li>Quite new in infancy state, yet to be tested in big companies<\/li>\n<li>Not for heavy lifting work like Spark Streaming, Flink<\/li>\n<\/ul>\n<p><a href=\"https:\/\/www.linkedin.com\/pulse\/spark-streaming-vs-flink-storm-kafka-streams-samza-choose-prakash\">https:\/\/www.linkedin.com\/pulse\/spark-streaming-vs-flink-storm-kafka-streams-samza-choose-prakash<\/a><\/p>\n<p><a href=\"https:\/\/www.confluent.io\/blog\/introducing-kafka-streams-stream-processing-made-simple\/\">https:\/\/www.confluent.io\/blog\/introducing-kafka-streams-stream-processing-made-simple\/<\/a><\/p>\n<h3>The Connector API<\/h3>\n<p>Connector API\ub97c \uc774\uc6a9\ud558\uba74 Producer Application \uacfc \uac19\uc774 \uc678\ubd80 \uc694\uc18c\ub97c \uc0ac\uc6a9\ud558\uc9c0 \uc54a\uace0 \ub370\uc774\ud130\ub97c \uac00\uc838\uc624\uace0, Consumer Apllication \uacfc \uac19\uc774 PULL \uc694\uccad\uc5d0 \ub370\uc774\ud130\ub97c \uc8fc\ub294 \uac83\uc774 \uc544\ub2c8\ub77c \uc8fc\uae30\uc801\uc73c\ub85c PUSH \ud558\ub294 \uad6c\uc870\ub3c4 \uac00\ub2a5\ud55c \uac83 \uac19\ub2e4.<\/p>\n<ul>\n<li>Simplifies data flow in to and out of Kafka<\/li>\n<li>Reduces development costs compared to custom connectors\n<ul>\n<li>Integrated schema management<\/li>\n<li>Task partitioning and rebalancing<\/li>\n<li>offset management<\/li>\n<li>fault tolerance<\/li>\n<li>Delivery semantics, Operations, And Monitoring<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><a href=\"https:\/\/www.confluent.io\/product\/connectors\/\">https:\/\/www.confluent.io\/product\/connectors\/<\/a><\/p>\n<p><a href=\"https:\/\/www.slideshare.net\/KaufmanNg\/data-pipelines-with-kafka-connect\">https:\/\/www.slideshare.net\/KaufmanNg\/data-pipelines-with-kafka-connect<\/a><\/p>\n<h2>Topics and Logs<\/h2>\n<h3>Topic<\/h3>\n<blockquote><p>\n  \uce74\ud504\uce74\uc5d0 \ubcf4\uad00\ub418\ub294 \uba54\uc2dc\uc9c0\ub97c \ubd84\ub958\ud558\ub294 \uae30\uc900<br \/>\n  Producer\uc640 Consumer\ub97c \uc5f0\uacc4\ud558\ub294 Channel\n<\/p><\/blockquote>\n<p>topic\uc740 \uc2a4\ud2b8\ub9bc \ub370\uc774\ud130\ub97c \ubd84\ub958\ud558\ub294 category \ub610\ub294 feed name\uc774\ub2e4. \ud558\ub098 \uc774\uc0c1\uc758 consumer \ub610\ub294 consumer group \uc744 \uac00\uc9c8 \uc218 \uc788\uace0, consumer group \ubcc4\ub85c \ubcc4\ub3c4\uc758 offset\uc744 \uad00\ub9ac\ud558\ubbc0\ub85c consumer group\uc774 \uc5ec\ub7ff\uc774\ub354\ub77c\ub3c4 \ubaa8\ub450 \uac19\uc740 \ub370\uc774\ud130\ub97c \ubcfc \uc218 \uc788\ub2e4.<\/p>\n<h4>@LinkedIn<\/h4>\n<p>5000\uac1c \uc774\uc0c1\uc758 topic \uc744 \ud558\ub098\uc758 \ud074\ub7ec\uc2a4\ud130\uc5d0\uc11c \ucc98\ub9ac\ud558\uace0 \uc788\ub2e4.<\/p>\n<p>https:\/\/stackoverflow.com\/questions\/32950503\/can-i-have-100s-of-thousands-of-topics-in-a-kafka-cluser<\/p>\n<h4>how many topics can kafka support?<\/h4>\n<ul>\n<li>it depends:\n<ul>\n<li>num of dirs allowed in FS<\/li>\n<li>open file handlers (kafka keep all log segments open in the broker)<\/li>\n<li>ZK nodes<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h3>Partitioning<\/h3>\n<p>kafka \ud074\ub7ec\uc2a4\ud130\ub294 \uc544\ub798\uc640 \uac19\uc774 \ud558\ub098\uc758 \ud1a0\ud53d\uc5d0 \ub300\ud574 \ud30c\ud2f0\uc154\ub2dd\uc744 \ud560 \uc218 \uc788\ub294\ub370,<\/p>\n<p><img decoding=\"async\" src=\"http:\/\/kafka.apache.org\/11\/images\/log_anatomy.png\" alt=\"Anatomy of a Topic\" \/><\/p>\n<p>Producer\uac00 \uba54\uc2dc\uc9c0\ub97c \uc2e4\uc81c\ub85c \uc5b4\ub5a4 partition\uc73c\ub85c \uc804\uc1a1\ud560\uc9c0\ub294 \uc0ac\uc6a9\uc790\uac00 \uad6c\ud604\ud55c partition \ubd84\ubc30 \uc54c\uace0\ub9ac\uc998\uc5d0 \uc758\ud574 \uacb0\uc815\ub41c\ub2e4. \uc608\ub97c \ub4e4\uc5b4 \ub77c\uc6b4\ub4dc-\ub85c\ube48 \ubc29\uc2dd\uc758 partition \ubd84\ubc30 \uc54c\uace0\ub9ac\uc998\uc744 \uad6c\ud604\ud558\uc5ec \uac01 partition\uc5d0 \uba54\uc2dc\uc9c0\ub97c \uade0\ub4f1\ud558\uac8c \ubd84\ubc30\ud558\ub3c4\ub85d \ud558\uac70\ub098, \uba54\uc2dc\uc9c0\uc758 \ud0a4\ub97c \ud65c\uc6a9\ud558\uc5ec \uc54c\ud30c\ubcb3 A\ub85c \uc2dc\uc791\ud558\ub294 \ud0a4\ub97c \uac00\uc9c4 \uba54\uc2dc\uc9c0\ub294 P0\uc5d0\ub9cc \uc804\uc1a1\ud558\uace0, B\ub85c \uc2dc\uc791\ud558\ub294 \ud0a4\ub97c \uac00\uc9c4 \uba54\uc2dc\uc9c0\ub294 P1\uc5d0\ub9cc \uc804\uc1a1\ud558\ub294 \ud615\ud0dc\uc758 \uad6c\uc131\ub3c4 \uac00\ub2a5\ud558\ub2e4.<\/p>\n<ul>\n<li>\ud30c\ud2f0\uc158 \ubc29\uc2dd\n<ul>\n<li>RDBMS\uc758 Partition \ucc98\ub7fc Key\ub97c \ud1b5\ud574 \ubd84\ub9ac<\/li>\n<li>Event \ubc1c\uc0dd \uc2dc\uc810 \uae30\uc900\uc73c\ub85c Round-robin<\/li>\n<li>modulo \uc5f0\uc0b0<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>\uc880 \ub354 \ubcf5\uc7a1\ud55c \uc608\ub85c\uc368 \uc0ac\uc6a9\uc790 ID\uc758 CRC32\uac12\uc744 partition\uc758 \uc218\ub85c modulo \uc5f0\uc0b0\uc744 \uc218\ud589\ud558\uc5ec(CRC32(ID) % partition\uc758 \uc218) \ub3d9\uc77c\ud55c ID\uc5d0 \ub300\ud55c \uba54\uc2dc\uc9c0\ub294 \ub3d9\uc77c\ud55c partition\uc5d0 \ud560\ub2f9\ub418\ub3c4\ub85d \uad6c\uc131\ud560 \uc218\ub3c4 \uc788\ub2e4\uace0 \ud55c\ub2e4.<\/p>\n<p>\uadf8\ub7f0\ub370 \uce74\uce74\uc624 Dev\uc5d0\uc11c Partition \ub098\ub204\uba74 \uc21c\uc11c \ubcf4\uc7a5\uc774 \uc548\ub420\uc218\ub3c4 \uc788\ub2e4\uace0 \ud558\ub358\ub370 \uad73\uc774 \ud30c\ud2f0\uc154\ub2dd\uc744 \ud558\ub294 \uc774\uc720\ub294?<\/p>\n<blockquote><p>\n  \uac01 \ud30c\ud2f0\uc158\uc758 \ub9ac\ub354\uac00 \uc5ec\ub7ec \ube0c\ub85c\ucee4\uc5d0 \ub098\ub204\uc5b4\uc838 \ubd84\ud3ec\ud558\ub3c4\ub85d \uc54c\uace0\ub9ac\uc998\uc774 \ub3fc \uc788\uc5b4 I\/O\ubd84\uc0b0\uc774 \ub41c\ub2e4. Consumer Group \uc744 \ud65c\uc6a9\ud558\ub294 \uacbd\uc6b0 \uba64\ubc84\ub4e4\uc774 \ud30c\ud2f0\uc158\uc744 \ub098\ub204\uc5b4 \uac00\uc838\uac08 \uc218 \uc788\uae30 \ub54c\ubb38\uc5d0 \uc758\ubbf8\uac00 \ub354 \uc0dd\uae30\ub294 \uac83 \uac19\ub2e4.\n<\/p><\/blockquote>\n<p><img decoding=\"async\" src=\"http:\/\/kafka.apache.org\/11\/images\/consumer-groups.png\" alt=\"consumer-groups\" \/><\/p>\n<p>\uc704\uc640 \uac19\uc774 Consumer Group A,B \uac00 \uc788\uc744 \ub54c A,B \ub294 \uac01\uc790 offset\uc744 \uad00\ub9ac\ud558\uc5ec topic\uc744 \uad6c\ub3c5\ud55c\ub2e4. \uadf8\ub7f0\ub370 Consuer Group A \ub0b4\uc5d0\uc11c\ub294 Consumer C1,C2 \uac00 P0,P1,P2,P3 \ub97c \ub098\ub204\uc5b4\uc11c \ubd80\ubd84\uc801\uc73c\ub85c \uad6c\ub3c5\ud55c\ub2e4.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/t1.daumcdn.net\/cfile\/tistory\/2655FB425509181D07\" alt=\"broker_leader\" \/><\/p>\n<p>\uc774 \ub54c\uc5d0 \uac01 \ud30c\ud2f0\uc158\ubcc4 leader P0,P1,P2,P3 \ub294 \uc5ec\ub7ec \ube0c\ub85c\ucee4 \uc11c\ubc84\uc5d0 \ubd84\uc0b0\ub418\uc5b4 \uc788\ub294\ub370 \uc774 \ub54c\ubb38\uc5d0 I\/O \uac00 \ubd84\uc0b0\ub418\uc5b4 \ud6a8\uc728\uc801\uc774\ub2e4.<\/p>\n<h2>Reliability<\/h2>\n<h3>ZooKeeper\ub97c \uc774\uc6a9\ud55c Clustering<\/h3>\n<p>\ub2e8\uc21c \uc218\uc9d1\uae30\ub9cc \uc0ac\uc6a9\ud588\uc744 \ub54c\uc758 \ubb38\uc81c\ub97c \ud574\uacb0\ud574 \uc90c. \uc11c\ube44\uc2a4 \uc911\ub2e8\uc774 \uc5c6\ub2e4.<\/p>\n<p><img decoding=\"async\" src=\".\/images\/kafka_cluster.png\" alt=\"cluster\" \/><\/p>\n<h3>ISR<\/h3>\n<blockquote><p>\n  In-Sync Replication (ISR) \ubaa8\ub378 \uc774\uc6a9\n<\/p><\/blockquote>\n<p>\ud30c\ud2f0\uc158 \ub2e8\uc704\ub85c replication\uc774 \uc0dd\uc131\ub418\ub294\ub370 \uc774 \ubd84\uc0b0\ub41c replication\uacfc primary \ub370\uc774\ud130(leader\uc640 \ubaa8\ub4e0 followers)\ub97c \uadf8\ub8f9\uc73c\ub85c \ubb36\uc74c.<\/p>\n<p>follower \ub4e4\uc740 leader \uc5d0\uac8c \uac31\uc2e0\ub41c \ub370\uc774\ud130\uac00 \uc788\ub0d0 \uc9c0\uc18d\uc801\uc73c\ub85c \uc694\uccad\ud558\uc5ec \uac31\uc2e0\ub41c \uac83\uc774 \uc788\uc73c\uba74 \ub370\uc774\ud130\ub97c \ubc1b\uc544\uac00\ub294 (Pulling) \ubc29\uc2dd\ub85c \ub370\uc774\ud130\ub97c \ubcf5\uc81c\ud558\ub294\ub370, \uc5ec\uae30\uc11c \uac31\uc2e0 \ub370\uc774\ud130 \uc694\uccad\uc744 \uc77c\uc815 \uc2dc\uac04\ub3d9\uc548 \ud558\uc9c0 \uc54a\ub294 follower\uac00 \uc788\uc73c\uba74 \uc774 follower\ub97c ISR\uc5d0\uc11c \uc81c\uc678\uc2dc\ud0a8\ub2e4. \ub610\ud55c \ud574\ub2f9 follower\uac00 \uae30\ub85d \ud588\uc5c8\ub358 offset\uc744 High watermark \ucc98\ub9ac \ud558\uc5ec \ub514\uc2a4\ud06c\uc5d0 \uc368\ub45a\uc73c\ub85c\uc368 \uc774\ud6c4 \uc7a5\uc560\uac00 \ub0ac\ub358 follower\uac00 \uc0b4\uc544\ub098\uba74 hw \ub9c8\ud06c\ubd80\ud130 \ub370\uc774\ud130\ub97c \uac00\uc838\uac08 \uc218 \uc788\ub3c4\ub85d\ud560 \uc218 \uc788\ub2e4.<\/p>\n<p><a href=\"https:\/\/engineering.linkedin.com\/kafka\/intra-cluster-replication-apache-kafka\">https:\/\/engineering.linkedin.com\/kafka\/intra-cluster-replication-apache-kafka<\/a><\/p>\n<p><img decoding=\"async\" src=\"https:\/\/content.linkedin.com\/content\/dam\/engineering\/en-us\/blog\/migrated\/kafka_replication_diagram.png\" alt=\"ISR\" \/><\/p>\n<p><img decoding=\"async\" src=\".\/images\/kafka_ISR.png\" alt=\"ISR\" \/><\/p>\n<hr \/>\n","protected":false},"excerpt":{"rendered":"<p>Apache Kafka Introduction \uc774\ubc88\uc5d0 \uc9c1\uc811 \uc0ac\uc6a9\ud574\ubcf4\ub2c8.. Message Queue Open Source Framework\ub85c\uc11c kafka\ub97c \ub9ce\uc774\ub4e4 \uc0ac\uc6a9\ud558\ub294 \uac83 \uac19\ub2e4. \uae30\ubcf8\uc801\uc73c\ub85c Producer\/Consumer Application \uc744 \uac1c\ubc1c\ud574\uc11c \uc0ac\uc6a9\ud574\uc57c \ud558\ub294\ub370 \ub9ce\uc774 \uc0ac\uc6a9\ud558\ub2e4 \ubcf4\ub2c8 \uad00\ub828 \uc790\ub8cc\uac00 \ub9ce\uace0 Hadoop, Spark, \uc5ec\ub7ec \uc218\uc9d1\uae30 \ub4f1\uc5d0\uc11c Plugin\uc744 \uc81c\uacf5\ud55c\ub2e4. \ub9ce\uc774 \uc4f0\uace0 \uc790\ub8cc\uac00 \ub9ce\uc544\uc11c kafka\ub97c \uc774\uc6a9\ud558\ub294 \uac83\ub3c4 \uc88b\uc740 \uc774\uc720\uc778 \uac83 \uac19\ub2e4. \ub2e4\uc74c \ub450\uac00\uc9c0 \uc774\uc720\uac00 \uc9c1\uc811 \ub290\ub080 \uc88b\uc740 \uc810 Consumer\ub294 [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[16],"tags":[139],"class_list":["post-527","post","type-post","status-publish","format-standard","hentry","category-kafka","tag-kafka"],"_links":{"self":[{"href":"https:\/\/oboki.net\/workspace\/wp-json\/wp\/v2\/posts\/527","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/oboki.net\/workspace\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/oboki.net\/workspace\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/oboki.net\/workspace\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/oboki.net\/workspace\/wp-json\/wp\/v2\/comments?post=527"}],"version-history":[{"count":4,"href":"https:\/\/oboki.net\/workspace\/wp-json\/wp\/v2\/posts\/527\/revisions"}],"predecessor-version":[{"id":1235,"href":"https:\/\/oboki.net\/workspace\/wp-json\/wp\/v2\/posts\/527\/revisions\/1235"}],"wp:attachment":[{"href":"https:\/\/oboki.net\/workspace\/wp-json\/wp\/v2\/media?parent=527"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/oboki.net\/workspace\/wp-json\/wp\/v2\/categories?post=527"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/oboki.net\/workspace\/wp-json\/wp\/v2\/tags?post=527"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}