210 Life

Hadoop과 Spark는 빅데이터 처리를 위한 오픈 소스 소프트웨어 프레임워크로, 
대용량 데이터를 저장하고 처리하기 위한 기술을 제공한다.

Hadoop:

Hadoop은 아파치 재단에서 개발한 분산 데이터 처리 프레임워크이며, 
주요 구성 요소로는 Hadoop Distributed File System (HDFS)와 MapReduce가 있다.
HDFS: 대용량 데이터를 여러 노드에 분산하여 저장하는 파일 시스템이다. 
데이터를 여러 블록으로 나눠 분산 저장하고, 
데이터의 복제를 통해 내결함성을 제공한다.
MapReduce: 데이터를 처리하는 분산 컴퓨팅 모델로, 대규모 데이터셋을 여러 노드에서 처리하고 결과를 집계한다. 주로 일괄 처리 작업에 사용된다.
Hadoop은 대규모 배치 작업에 적합하며, 안정성과 내결함성이 중요한 환경에서 사용된다. 다만, 실시간 데이터 처리와 반응성면에서는 제한이 있다.


Apache Spark:

Apache Spark은 빅데이터 처리를 위한 빠르고 다목적 분산 처리 프레임워크이며, 
빅데이터 처리 작업을 빠르게 수행할 수 있으며, 다양한 데이터 처리 작업을 지원한다.
Spark는 메모리 기반 데이터 처리를 사용하여 디스크 기반의 Hadoop MapReduce보다 훨씬 빠른 처리 속도를 제공한다. 
또한, 다양한 데이터 소스와 통합이 쉽고 다양한 데이터 처리 라이브러리를 포함한다.
Spark는 배치 처리, 스트리밍 데이터 처리, 머신러닝 및 그래프 처리 등 다양한 작업을 처리할 수 있다.
스트리밍 처리를 지원하므로 실시간 데이터 처리도 가능하다.
Spark은 다양한 언어에서 사용 가능하며, 대부분의 언어로 개발된 라이브러리를 사용할 수 있다.

요약하면, 
Hadoop은 
대용량 배치 처리와 데이터 저장을 위한 프레임워크로, 
안정성과 내결함성을 강조한다. 
반면에 Spark는 
빠르고 다목적 데이터 처리를 지원하며, 
실시간 및 배치 처리를 모두 다루는데 적합하다. 

두 기술은 서로 보완적으로 사용되기도 하며, 사용 사례와 요구 사항에 따라 선택된다.

XML과 JSON은 모두 데이터를 표현하고 교환하기 위한 형식이며,
각각의 특징과 장단점운 아래 와 같다.

**XML (Extensible Markup Language):**
- **장점:**
  1. **계층 구조**: XML은 계층 구조를 가짐으로써 복잡한 데이터 구조를 표현하는 데 용이하다.
  2. **설명력**: XML 태그는 데이터에 대한 설명을 포함할 수 있으며, 이로 인해 데이터 의미를 이해하기 쉬워진다.
  3. **DTD 및 XML 스키마**: DTD 및 XML 스키마를 사용하여 데이터의 유효성을 검사하고 데이터 일관성을 보장할 수 있다.
  4. **다양한 문자 인코딩**: XML은 다양한 문자 인코딩 방식을 지원하므로 국제화 및 지역화에 유용하다.

- **단점:**
  1. **불필요한 중복**: XML 문서는 종종 반복적인 태그와 불필요한 길이를 가질 수 있어 데이터 크기가 상대적으로 크다.
  2. **구문 복잡성**: XML 문서는 파싱 및 생성이 JSON에 비해 복잡하고 느릴 수 있다.
  
**JSON (JavaScript Object Notation):**
- **장점:**
  1. **간결함**: JSON은 간단한 데이터 구조로 데이터를 표현한다. 이로써 데이터 크기가 작아지고 처리가 빠르다.
  2. **가독성**: JSON은 데이터를 사람이 읽기 쉬운 형식으로 표현하며, 프로그래머들에게 친숙하다.
  3. **빠른 파싱**: JSON 문서는 파싱이 빠르며 대부분의 프로그래밍 언어에서 지원한다.
  4. **JavaScript 통합**: JSON은 JavaScript와 연동이 용이하며, 웹 애플리케이션에서 일반적으로 사용된다.

- **단점:**
  1. **계층 구조 제한**: JSON은 XML보다 덜 계층적이며 복잡한 데이터 구조를 나타내기 어려울 수 있다.
  2. **설명력 부족**: JSON은 태그에 대한 설명을 제공하지 않으며, 데이터 의미를 파악하기 어려울 수 있다.

간단한 데이터 교환 및 웹 서비스 통신에는 JSON이 효율적이며, 
복잡한 계층 구조 및 설명력이 필요한 경우에는 XML을 고려할 수 있다. 
JSON과 XML을 혼합하여 사용하거나, 데이터 변환 도구를 사용하여 두 형식 간에 데이터를 변환하기도 한다.

XML(Extensible Markup Language)은 
데이터를 저장하고 전송하기 위한 마크업 언어의 한 형식이며, 
사람과 기계가 모두 이해하기 쉽고, 
다양한 종류의 데이터를 구조화하고, 
교환하기 위한 강력한 도구로 사용된다. 
주요 특징과 사용 사례로는,

1. **마크업 언어**:
   XML은 다른 데이터를 표현하는 데 사용되는 마크업 언어이다. 
     태그를 사용하여 데이터 요소를 정의하고 감싸며, 
     각 요소는 시작 태그(`<태그이름>`)와 종료 태그(`</태그이름>`)로 둘러싸여 있다.

2. **계층 구조**:
   XML 데이터는 계층 구조를 가진다다.
   각 요소는 다른 요소 내부에 중첩될 수 있으며, 
   이로 인해 데이터의 계층 구조를 잘 표현할 수 있다.

3. **유연성**:
   XML은 사용자가 자신만의 태그와 데이터 구조를 정의할 수 있는 확장 가능한 언어이다. 
   이는 다양한 도메인과 응용 분야에 맞게 데이터를 정의할 수 있는 장점을 갖고 있다.

4. **플랫폼 독립성**:
   XML은 플랫폼 독립적이며, 
   서로 다른 시스템 및 프로그래밍 언어 간에 데이터 교환을 쉽게 할 수 있도록 해준다.

5. **문서 기반 데이터 저장**:
   XML은 주로 구조화된 데이터를 저장하고 전송하기 위해 사용된다. 
   예를 들어, 구성 파일, 웹 서비스 응답, 설정 데이터, 인터넷에서 데이터 교환 등에 활발하게 사용된다.

6. **웹 서비스와 데이터 교환**:
   XML은 웹 서비스의 표준 데이터 형식 중 하나로 사용되며, 
   데이터를 클라이언트 및 서버 간에 교환할 때 주로 XML을 사용한다.

7. **DTD 및 스키마**:
   XML 문서의 구조를 정의하기 위해 Document Type Definition(DTD) 또는 XML 스키마를 사용할 수 있다. 
   이러한 정의는 XML 문서의 유효성을 검사하고 데이터의 일관성을 보장하는 데 도움이 된다.

8. **Well-Formed vs. Valid XML**:
   Well-Formed XML 문서는 XML의 기본 규칙을 준수하는 문서를 말하며, 
   Valid XML 문서는 DTD 또는 XML 스키마와 같은 정의를 따르는 문서를 말한다.

f = open(“test.txt”, ‘w’)
for j in range(0, 11): # 0 부터 시작해서 11개니까 0~10이다
   for i in range(0, 6): # 0 부터 시작해서 6개니까 0~5이다
    pixelx[j][i] = cX * i + j
      data1 = str(pixelx[j][i]) + '\t' 
      file1.write(data1)
   data1 = '\n'
   file1.write(data1)
file1.close()
file11 = open("./output/STDx.txt", 'r')
read = file11.read()
print(read)
======
결과
0       1       2       3       4       5
1       2       3       4       5       6
2       3       4       5       6       7
3       4       5       6       7       8
4       5       6       7       8       9
5       6       7       8       9       10
6       7       8       9       10      11
7       8       9       10      11      12
8       9       10      11      12      13
9       10      11      12      13      14
10      11      12      13      14      15

import numpy as np

import json

#다음과 같이 2차배열에 각각의 값을 저장한 후에

pixelx = [[0 for col in range(6)] for row in range(11)]

cX = 1

for j in range(0, 11): # 0 부터 시작해서 11개니까 0~10이다

     for i in range(0, 6): # 0 부터 시작해서 6개니까 0~5이다

          pixelx[j][i] = cX * i + j

with open('./array2.json', "w") as file:

     json.dump(pixelx, file)

file.close()

with open('./array2.json', "r") as file:

     data = json.load(file)

file.close()

print(pixelx)

print(data)

print("data[0][0] = ", data[0][0])

print("data[0][1] = ", data[0][1])

print("data[3][3] = ", data[3][3])

==================================

실행결과

[[0, 1, 2, 3, 4, 5], [1, 2, 3, 4, 5, 6], [2, 3, 4, 5, 6, 7], [3, 4, 5, 6, 7, 8], [4, 5, 6, 7, 8, 9], [5, 6, 7, 8, 9, 10], [6, 7, 8, 9, 10, 11], [7, 8, 9, 10, 11, 12], [8, 9, 10, 11, 12, 13], [9, 10, 11, 12, 13, 14], [10, 11, 12, 13, 14, 15]]

[[0, 1, 2, 3, 4, 5], [1, 2, 3, 4, 5, 6], [2, 3, 4, 5, 6, 7], [3, 4, 5, 6, 7, 8], [4, 5, 6, 7, 8, 9], [5, 6, 7, 8, 9, 10], [6, 7, 8, 9, 10, 11], [7, 8, 9, 10, 11, 12], [8, 9, 10, 11, 12, 13], [9, 10, 11, 12, 13, 14], [10, 11, 12, 13, 14, 15]]

data[0][0] = 0

data[0][1] = 1

data[3][3] = 6

=================================================

array2.json파일 저장내용

[[0, 1, 2, 3, 4, 5], [1, 2, 3, 4, 5, 6], [2, 3, 4, 5, 6, 7], [3, 4, 5, 6, 7, 8], [4, 5, 6, 7, 8, 9], [5, 6, 7, 8, 9, 10], [6, 7, 8, 9, 10, 11], [7, 8, 9, 10, 11, 12], [8, 9, 10, 11, 12, 13], [9, 10, 11, 12, 13, 14], [10, 11, 12, 13, 14, 15]]

파이썬에서 객체를 바이트 변환하고 다시객체로 복원하는 것을 직렬화(Serialization), 이렇게 파이썬 객체를 일련의 바이트들로 변환하는 것을 직렬화(Serialization)라 하고, 다시 바이트들을 객체로 변환하는 것을 역직렬화(Deserialization)이라 한다.

파이썬에서 이와 같은 직렬화, 반직렬화는 pickle, cPickle을 사용할 수 있다.

gray=cv2.threshold(gray, 0, 255, cv2.THRESH_OTSU )[1] 

이진화 처리는 간단하지만, 쉽지 않은 문제를 가지고 있다. 
이진화란 영상을 흑/백으로 분류하여 처리하는 것을 말합니다. 
이때 기준이 되는 임계값을 어떻게 결정할 것인지가 중요한 문제가 됩니다. 
임계값보다 크면 백, 작으면 흑이 됩니다. 
기본 임계처리는 사용자가 고정된 임계값을 결정하고 그 결과를 보여주는 단순한 형태입니다.

이때 사용하는 함수가 cv2.threshold() 입니다.

cv2.threshold(src, thresh, maxval, type) → retval, dst
Parameters:
src – input image로 single-channel 이미지.(grayscale 이미지)
thresh – 임계값
maxval – 임계값을 넘었을 때 적용할 value
type – thresholding type
thresholding type은 아래와 같습니다.
cv2.THRESH_BINARY
cv2.THRESH_BINARY_INV
cv2.THRESH_TRUNC
cv2.THRESH_TOZERO
cv2.THRESH_TOZERO_INV
아래 예제는 각 type별 thresholding 결과입니다.

Sample Code

import cv2
import numpy as np
from matplotlib import pyplot as plt

img = cv2.imread('gradient.jpg',0)

ret, thresh1 = cv2.threshold(img,127,255, cv2.THRESH_BINARY)
ret, thresh2 = cv2.threshold(img,127,255, cv2.THRESH_BINARY_INV)
ret, thresh3 = cv2.threshold(img,127,255, cv2.THRESH_TRUNC)
ret, thresh4 = cv2.threshold(img,127,255, cv2.THRESH_TOZERO)
ret, thresh5 = cv2.threshold(img,127,255, cv2.THRESH_TOZERO_INV)

titles =['Original','BINARY','BINARY_INV','TRUNC','TOZERO','TOZERO_INV']
images = [img,thresh1,thresh2,thresh3,thresh4,thresh5]

for i in xrange(6):
plt.subplot(2,3,i+1),plt.imshow(images[i],'gray')
plt.title(titles[i])
plt.xticks([]),plt.yticks([])

plt.show()

여러 이미지를 하나의 화면에 보여줄때 plt.subplot() 함수를 사용합니다. 사용법은 위 소스나 Matplotlib Document를 참고하시기 바랍니다.

https://opencv-python.readthedocs.io/en/latest/doc/09.imageThresholding/imageThresholding.html

색상 공간 변환(Convert Color)은 본래의 색상 공간에서 다른 색상 공간으로 변환할 때 사용한다.
색상 공간 변환 함수는 데이터 타입을 같게 유지하고 채널을 변환한다.
입력된 이미지는 8 비트, 16 비트, 32 비트의 정밀도를 갖는 배열을 사용할 수 있다.
출력된 이미지는 입력된 이미지의 이미지 크기와 정밀도가 동일한 배열이 된다.
채널의 수가 감소하게 되어 이미지 내부의 데이터는 설정한 색상 공간과 일치하는 값으로 변환되며, 데이터 값이 변경되거나 채널 순서가 변경될 수 있다.

색상 공간 변환(Convert Color)은 본래의 색상 공간에서 다른 색상 공간으로 변환할 때 사용합니다.
색상 공간 변환 함수는 데이터 타입을 같게 유지하고 채널을 변환합니다.
입력된 이미지는 8 비트, 16 비트, 32 비트의 정밀도를 갖는 배열을 사용할 수 있습니다.
출력된 이미지는 입력된 이미지의 이미지 크기와 정밀도가 동일한 배열이 됩니다.
채널의 수가 감소하게 되어 이미지 내부의 데이터는 설정한 색상 공간과 일치하는 값으로 변환되며, 데이터 값이 변경되거나 채널 순서가 변경될 수 있습니다.

메인코드
import cv2

src = cv2.imread("Image/crow.jpg", cv2.IMREAD_COLOR)
dst = cv2.cvtColor(src, cv2.COLOR_BGR2GRAY)

cv2.imshow("src", src)
cv2.imshow("dst", dst)
cv2.waitKey()
cv2.destroyAllWindows()

세부코드
dst = cv2.cvtColor(src, cv2.COLOR_BGR2GRAY)


색상 공간 변환 함수(cv2.cvtcolor)로 이미지의 색상 공간을 변경할 수 있습니다.
dst = cv2.cvtcolor(src, code, dstCn)는 입력 이미지(src), 색상 변환 코드(code), 출력 채널(dstCn)으로 출력 이미지(dst)을 생성합니다.
색상 변환 코드는 원본 이미지 색상 공간2결과 이미지 색상 공간을 의미합니다.
원본 이미지 색상 공간은 원본 이미지와 일치해야합니다.
출력 채널은 출력 이미지에 필요한 채널의 수를 설정합니다.
Tip : BGR은 RGB 색상 채널을 의미합니다. (Byte 역순)
Tip : 출력 채널은 기본값을 사용하여 자동으로 채널의 수를 결정하게 합니다.


추가 정보Permalink
채널 범위Permalink
형식 범위
CV_8U 0 ~ 255
CV_16U 0 ~ 65535
CV_32F 0 ~ 1


색상 공간 코드Permalink
속성 의미 비고
BGR Blue, Green, Red 채널 -
BGRA Blue, Green, Red, Alpha 채널 -
RGB Red, Green, Blue 채널 -
RGBA Red, Green, Blue, Alpha 채널 -
GRAY 단일 채널 그레이스케일
BGR565 Blue, Green, Red 채널 16 비트 이미지
XYZ X, Y, Z 채널 CIE 1931 색 공간
YCrCb Y, Cr, Cb 채널 YCC (크로마)
HSV Hue, Saturation, Value 채널 색상, 채도, 명도
Lab L, a, b 채널 반사율, 색도1, 색도2
Luv L, u, v 채널 CIE Luv
HLS Hue, Lightness, Saturation 채널 색상, 밝기, 채도
YUV Y, U, V 채널 밝기, 색상1, 색상2
BG, GB, RG 디모자이킹 단일 색상 공간으로 변경
_EA 디모자이킹 가장자리 인식
_VNG 디모자이킹 그라데이션 사용


원본 이미지 색상 공간2결과 이미지 색상 공간에 색상 공간 코드를 조합하여 사용할 수 있습니다.

예) BGR2GRAY는 Blue, Green, Red 채널 이미지를 단일 채널, 그레이스케일 이미지로 변경합니다.

https://076923.github.io/posts/Python-opencv-10/#%EC%84%B8%EB%B6%80-%EC%BD%94%EB%93%9C

워싱턴포스트의 집계에 따르면 미국에서는 작년에만 1천42명이 근무 중인 경찰의 총에 맞아 숨졌다.

이러한 사망자 중 절반은 백인이지만 인구당 비율을 비교하면 흑인이 더 많이 죽는 셈이다.

흑인은 미국 전체 인구의 13%를 차지하지만 경찰에 목숨을 잃는 비율은 백인보다 2배 이상 높다.

백인 사망자는 100만 명 당 16명이지만 흑인 사망자는 100만 명 당 40명이다. 미국 사회의 다른 유색인종 히스패닉도 100만 명 중 28명으로 백인보다 높게 나타났다.

1. Don’t Remove Grass Clippings.
Leaving grass clippings on the lawn after you mow can cause thatch problems, right? Nope! That’s a myth. Turns out, grass clippings can actually help with the overall health of your lawn. And that means less work for you when you mow.

2. Don’t Water Every Day
Did you know your lawn can actually become dependent and needy if it has too much water? Instead of watering every day for 15 minutes, choose one day a week to water the lawn for an entire hour. Deep watering makes your lawn healthier and more drought-tolerant. Take a look at these essential things you should do to your lawn right now.

3. Don’t Forget to De-Thatch
Thatch is a layer of slowly decomposing grass stems, roots, clippings and debris that accumulate at the soil surface over time. It can build up in your lawn and virtually choke it to death. If you’re wondering why your lawn isn’t healthy and lush, thatch buildup could be the answer.

Excessive thatch buildup is commonly found in lawns that have been over-fertilized or over-watered and have never been aerated. Thatch buildup of 3/4-in. or more will restrict water and nutrient penetration into the soil (think thatched roof) and can harbor disease-causing organisms that can increase the need for pesticides. Slice open a section of turf. If the thatch is more than 3/4-in. thick, take action.

4. Don’t Fertilize Shady Areas More
People tend to over-apply fertilizer to shady areas because the grass is struggling. But that just kills it faster!

Many people really have two lawns — the one that gets full sun for most of the day, and a shaded lawn that may get only two to four hours of direct sun. Their water and fertilizer needs differ. The shady-area grass needs less water because less evaporates, and less fertilizer because with less sun it doesn’t grow as much. When you go into shade, shift the controls on the spreader so you’re spreading about half the amount. Also, be sure to check that none of these invasive plants are in your backyard.

5. Don’t Give Up on Shady Areas
Growing grass under shade trees isn’t easy, but one key to success is choosing the right shade grass species and planting method for your region. In cool-season areas, you’ll get better results with seed than sod. Sod is grown in wide-open fields under conditions that favor sun-loving grasses.

Choose red and tall fescues for shady areas in Northern zones. Garden centers will have grass seed mixes formulated for shade. Late summer and mid-spring are the best times to establish cool-season grasses in shady areas. Check out these landscaping design trends you’ll probably see this year.

6. Don’t Forget to Check Soil Moisture
After an extended warm, dry period (dry soil is the key), set up your sprinkler and set a timer for 30 minutes. Then turn off the water and check the soil for moisture depth. Do this by pushing a shovel into the lawn and tipping it forward to expose the soil. See how deep the water has penetrated. Moist soil will be darker. Your goal is to run the sprinkler until the water penetrates three to four inches into the soil.

If the water has not penetrated far enough, restart the watering and continue to keep track of the time. Check again in another 15 minutes. With trial and error, you’ll eventually arrive at the optimal length of time to water for your soil type and water pressure.

7. Don’t Wait Too Long Between Mowing
If you came back from a vacation and your yard is a wilderness, don’t try and mow it all in one day. Cut off some of the length, then wait a couple of days and mow again. This will cause less stress on the grass. You may need three passes depending on how long the grass grew. These lawn and gardening tools are definitely worth the splurge.

8. Don’t Cut Grass Too Short
Every grass type has an optimal cutting height, and you’re better off on the high side of that height. There are a few reasons. Each grass blade is a food factory of the plant. Short blades can’t generate as much food as long blades. Long blades also shade and cool the soil. That means weed seeds are less likely to sprout, and you won’t have to water as often because water won’t evaporate as fast. Not sure what type of grass you have? Take a sample to a garden center for identification.

9. Don’t Mow in the Same Direction Every Time
It’s easy to fall into a routine as you mow your lawn week in, week out over the course of a summer. But try not to make your lawn mowing routine too repetitive. Instead, mow in a different direction every time: front to back, back to front, diagonal, etc. Repeatedly mowing the exact same way will cause the grass blades to grow at an angle, and you may develop permanent tracks from the mower wheels. Here are some easy ways to take your garden from good to great.

10. Don’t Cut Wet Grass
Mowing wet grass can cause the mower wheels to leave ruts in your yard, and you could leave behind giant clumps of clippings that could smother the grass beneath. And the wet grass will carpet the underside of your mower deck with a thick mat that’s a pain to clean. Here’s how to get rid of the worst garden pests.

11. Don’t Use Broadleaf Herbicides in Extreme Temperatures
You need to kill weeds when they’re growing. That’s because herbicides are absorbed through the leaves and then sent throughout the rest of the plant. When the weather is too cool, the weed isn’t growing and the herbicide won’t be absorbed, so the chemical isn’t as effective. Too hot, and the herbicide will stress the grass. The product directions will give you the best temperature range. Apply herbicides when rain isn’t forecasted; a soaking will just rinse off the herbicide before it can do any good.

12. Don’t Overfeed
If you apply too much grass fertilizer, especially in sandy soils, a good portion of it will leach through the soil and make its way into our precious groundwater, lakes, streams and wetlands. Lawn grasses only need a certain amount of food. More isn’t always better. We love these benefits of a healthy yard.

13. Don’t Ignore Pet Areas
Dog spots are round patches about four to eight inches in diameter with dead grass in the middle, encircled by dark green grass. Often caused by a winter’s worth of animal urine, they’re most apparent in the early spring when dormant grass first begins to turn green again. You have to replant your grass; it won’t come back on its own. But first you have to dilute or remove the caustic urine from the soil. Thoroughly soak the area with lots of water. Learn the next steps to repairing pet spots here.


14. Don’t Discount Compost
Top-dress your lawn with high-quality compost. Compost can bring depleted or damaged soil back to life, resulting in stronger root systems and happier plants. One teaspoon of compost contains a billion beneficial microorganisms that help create better soil structure and texture, which improves nutrient, water and air retention.

To apply compost, spread it over your lawn with a shovel, aiming for a layer 1/4- to 1/2-in. thick. Then work it into the turf with a rake. It’s best to do this after aerating. Most garden centers sell bagged compost. But to cover an entire yard, you’re better off buying in bulk. Don’t worry about buying too much—any leftovers will benefit your garden and shrub beds. By the way, here’s how to start composting at home

15. Don’t Remove ALL Fallen Leaves
Did you know that decomposing leaves are actually great for your lawn? Leaves have organic matter in them that works as a natural fertilizer, helping your grass to grow the following year.

According to Sam Bauer, a turfgrass researcher from the University of Minnesota, fallen leaves can even suppress the growth of weeds. He recommends mulching the leaves by using a mower (specifically with a specialized mulching blade, if you have one) to cut them up. However, if you have huge piles of leaves on your lawn, it may be hard to mulch all at once, and this can smother your grass. Remove those piles until you have a good dusting of leaves around your lawn before mulching with your mower.