
Donny
Hello, welcome to... Wait, stop clicking my avatar!
NLP with Deep Learning
2021-09-09
Machine Learning
332
Transformer
Attention is all you need - Arxiv
Feed Forward: Two fully connected layers (Linear) with a ReLU activation in between.
Multi-head Attention:
Attention
Attention - Qiita
Self-Attention
GPT (Generative Pre-Training)
GPT - paper
BERT (Bidirectional Encoder Representations from Transformers)
BERT - Arxiv
BERT Explained
Different from the Transformer (GPT) which is trained using only the left context, BERT uses bidirectional encoder which makes use of both the left and the right context. [MASK] is used to mask some of the words so that the model will not see the word itself indirectly.
Pre-training of BERT makes use of two strategies: MLM (Masked Language Model) and NSP (Next Sentence Prediction). The model is trained with both the strategies together.
As shown below, the input embeddings of BERT consists of the token embeddings, the segment embeddings, and the position embeddings. Note that a segment may consists of multiple sentences.
In MLM task,
Tic-Tac-Toe Online
2019-10-10
Tech
252
Tic-Tac-Toe Online Server
Base on the Tic-Tac-Toe Game of CS188, Berkeley, I develop an online version of Tic-Tac-Toe. Now your agent can play with my agent online!
I think it is a good way to check whether our agents are optimal or not.
My agent can beat random agents most of the time even if my agent is the second player.
Online Server Website: Tic-Tac-Toe Online
Download the attached client file from the moodle form and place it in the same directory of your solveTicTacToe.py (solveTicTacToe.py depends on util.py , so util.py is also needed).
Run this command:
$ python3 TicTacToeOLClient.py -u demo -n 3
And Enjoy!
Notice: You need to specify a username with "-u USERNAME". Don't use "demo" as your username cause it is forbidden.
Usage: TicTacToeOLClient.py [options]
Options
Variable
Description
-u
USERNAME
Username, must not be empty nor
Shenyang2019.icpc D Fish Eating Fruit
2019-09-18
ACM
419
The Question
Fish eating fruit on jisuanke
Given an undirected acyclic graph G, all possible path P in the graph, calculate:
The first taste
In the contest, a handsome foriegn teammate conviced me that this problem can be solve using LCA. I tried. And it did work, with the help of dijkstra.
My solution is to, first of all, run dijkstra, and get the distance between root node and every other nodes. Then, calculate the LCA for every two nodes. The desired result is:
It worked, but we got TLE for looping though all the nodes, which is .
The second trial
After the contest, I was told that this is a DP problem. You calculate the times an edge is accessed, times it with the weight, sum them up by the modulus of 3, you got the result.
This one, however, also got TLE. Oh, FISH!
The final solution
The reason why the second solution still can
R-CNN
2019-07-07
Computer Vision
2687
Overview
Regions with CNN features:
Efficient Graph Based Image Segmentation
use disjoint set to speed up merge operation
Selective Search
HOG (Histogram of Oriented Gradient)
Multiple criterions (color, texture, size, shape) to merge regions
AlexNet/VGG16
R-CNN
Notice that many descriptions are replicated from the orignal sources directly.
Some Fundermental Conceptions
Batch Size
Stochastic Gradient Descent. Batch Size = 1
Batch Gradient Descent. Batch Size = Size of Training Set
Mini-Batch Gradient Descent. 1 < Batch Size < Size of Training Set
Regularization
A regression model that uses L1 Regularization technique is called Lasso Regression and model which uses L2 is called Ridge Regression.
Ridge Regularization
Ridge regression adds "squared magnitude" of coefficient as penalty term to the loss function.
The first sum is an example of loss function.
Lasso Regularization
Lasso Regression (Least Absolute Shrinkage and Selection Operator) adds "absolute value of magnitude" of coefficient as penalty term to the loss function.
SIFT
2019-07-03
Computer Vision
389
Useful Materials
Distinctive Image Features from Scale-Invariant Keypoints[1] by David G. Lowe.
SIFT(Scale-Invariant Feature Transform)[2] on Towards Data Science.
The SIFT (Scale Invariant Feature Transform) Detector and Descriptor[3].
Notes
Uses DoG (Difference of Gaussian) to approximate Scale-normalized LoG (Laplacian of Gaussian)[4].
where is the two dimensions Gaussian function, and is the input image.
[need more consideration] After each octave, the Gaussian image is down-sampled by a factor of 2, by resampling the Gaussian image that has twice the initial value of by taking every second pixel in each row and column. And we start on the new octave with .
Since the image size is reduced to 1/4, the sigma for the next octave becomes , which is equal to .
To understand it, frist consider this question: If the image size is reduced to 1\4, but the kernel size of
Image Processing - Noise and denoise
2019-03-20
Computer Vision
455
Types of Noise
Additive noise
Additive noise is independent from image signal. The image g with nosie can be considered as the sum of ideal image f and noise n.[1]
Multiplicative noise
Multifplicative noise is often dependent on image signal. The relation of image and noise is[1]:
Gaussian noise
Gaussian noise, named after Carl Friedrich Gauss, is statistical noise having a probability density function (PDF) equal to that of the normal distribution, aka. the Gaussian distribution. i.e. the values that the noise can take on are Gaussian-distributed.
The PDF of a Gaussian random variable is given by[2]:
Salt-and-pepper noise
Fat-tail distributed or "impulsive" noise is sometimes called salt-and-pepper nosie or spike noise. An image containing salt-and-pepper noise will have dark pixels in bright regions and bright pixels in dark regions.[2]
The PDF of (Bipolar) Impulse noise is given by:
if b > a, gray-level
Greek Alphabet
2019-03-19
Stuff
81
Greek Alphabet
Letter (Capital Case)
Letter (Lower Case)
Name
English Equivalent
alpha
a
beta
b
gamma
g
delta
d
epsilon
e
zeta
z
eta
h
theta
th
iota
i
kappa
k
lambda
l
mu
m
nu
n
xi
x
omicron
o
pi
p
rho
r
sigma
s
tau
t
upsilon
u
phi
ph
chi
ch
psi
ps
omega
o
References
[1]. Greek Alphabet - Wikipedia
[2]. Greek alphabet letters & symbols
Decision Tree
2019-03-16
Machine-Learning
1037
Information Content
where,
I : the information content of X. An X with greater I value contains more information.
P : the probability mass function.
: b is the base of the logarithm used. Common values of b are 2 (bits), Euler's number e (nats), and 10 (bans).
Entropy (Information Theory)[1]
where,
H : the entropy H. Named after Boltzmann's H-theorem (but the definition is proposed by Shannon). H indicates the uncertainty of X.
P : probability mass function.
I : the information content of X.
E : the expected value operator.
The entropy can explicitly be written as:
ID3[2]
Use ID3 to build a decision tree:
Calculate the entropy of the samples under the current node.
Find a feature F that can maximize the information gain. The information gain is calculatd by:
where E is the entropy of
AutoTag
2019-03-12
Tech
364
AutoTag
AutoTag is a program that generate tags for documents automatically. The main process includes:
Participle (N-gram + lookup in dictionary)
Generate bag-of-words for each document.
Calculate term frequency and inverse document frequency.
Pick top x words with greater tf-idf values as tags.
N-gram
N-gram generate a sequence of n words in every position of a sentence.[1]
sentences = 'Lucy like to listen to music. Luna like music too.'
items = ngram(sentences, 2)
print(items)
# output:
[
'Lucy like',
'like to',
'to listen',
'listen to',
'to music',
'Luna like',
'like music',
'music too',
]
Bag of words
The bag-of-words model is a simplifying representation in NLP and IR.[1]
N-gram
Count the times that each word appears
w3m
2019-03-01
Tech
104
w3m: WWW wo Miru
(c) Copyright Akinori ITO
w3m is a pager with WWW capability. It IS a pager, but it can be used as a text-mode WWW browser.
Keyboard Shortcuts
Shortcut
Action
Level
H
Help
Brower
q
Quit w3m
Brower
C-h
History
Brower
T
New tab
Tabs
C-j
Open link on current tab
Tabs
C-t
Open link on new tab
Tabs
C-q
Close tab
Tabs
U
Go to url
Page
R
Reload
Page
B
Back
Page
Configuration
File
Location
Keymap file
~/.w3m/keymap
